Storj Labs - development largest blockchain file hosting

Storj Labs

Helping one of the world’s largest blockchain file hosting better serve their largest clients.

storj_labs_logo

The client

Storj is an open source, decentralized file storage solution. It uses encryption, file sharding, and a blockchain-based hash table to store files on a peer-to-peer network.

Starting out in US in 2011, Storj now has over ten million users, and serves tens of thousands of agencies and organizations.

Task

Storj is an open source, decentralized file storage solution. It uses encryption, file sharding, and a blockchain-based hash table to store files on a peer-to-peer network.

Starting out in US in 2011, Storj now has over ten million users, and serves tens of thousands of agencies and organizations.

Solution

We worked on a web app for a storage node operator dashboard, where node operators can check their statistics over the network and their payout information. We built a cloud storage management console. First we build OSS connector, created the first ever OS mirroring service that allows  users to migrate easily from one cloud provider to another, etc.

QA Front End Back End Product Design Software Engineering

What is Storj?

Storj is a blockchain-based cloud storage network which allows software developers to move application data on a decentralized network. The project was launched in 2014. The StorjcoinX (SJCX) token was issued through the Counterparty protocol, which runs on top of the Bitcoin blockchain. Later, the transfer of the token and the main architecture of the project to the Ethereum blockchain happened, since maintenance within the Bitcoin network was too expensive, and, most importantly, too slow. All the existing 500 million StorjcoinX coins were created in 2014 when the project was launched.

Storj is one of the successful implementations of the concept embodied in Permacoin. The concept of the Permacoin cryptocurrency was first proposed by five experts from the University of Maryland and the Cornell Tech Institute, who made up the Microsoft Research working group, back in 2010. Permacoin is a torrent-like protocol with an internal token that is used to reward users for storing data. 

The Storj platform can be considered an alternative to services such as Dropbox, and has certain analogies with torrents: information is divided into blocks, encrypted and distributed among the “farmers” – the project participants. The decentralized nature of the service, Storj Labs says, provides a higher level of security and information security.

case-2.png

The way Storj stores data is simple. A file uploaded by a user to the network is encrypted so that only the owner has access to it. The result is split into several parts and sent to the network. 

The process of providing disk space for storing data in exchange for project tokens is called farming. At an early stage of the development of the project, in order to start farming, you had to have 10,000 SJCX coins in your wallet, but later this rule was canceled.

Back in 2015, before the beta version of Storj was released a year later, the platform came to the attention of Dutch law enforcement agencies, who admitted that new generation technologies could significantly "make life difficult for them."

Australia has Wooranna Park Primary School, where children are taught the basics of distributed ledger technology from the first grade. The school even created its own cryptocurrency – Woorannacoin. Thanks to teaching in a playful way, complex technologies become understandable even for the youngest schoolchildren. In particular, one of the forms of training involves participation in the multiplayer game Minecraft. At the same time, the game server is connected to the Storj.io decentralized blockchain data storage.

ihnatuk
Viktor Ihnatiuk’s story about Fcs Ninja’ cooperation with Storj Labs
storj_labs_logo

Storj Mobile

We started working with Storj  at the end of 2017, and our first project was Storj Mobile. 

Storj Mobile is an application that is almost identical to the functionality of the Dropbox mobile application. In this application, you can create an account for yourself, create a private key to encrypt the data that you intend to store using this application. Also in the application you can create folders by uploading files to them – all these operations can be performed directly through the phone. Providing such an opportunity was quite difficult, since the phone is still weaker than the computer.

In this mobile application there is synchronization with different folders – in Android you can choose a customized number of folders that will sync with your Storj application. I mean, as soon as something is updated in this folder on the phone, this folder immediately "knocks" on the mobile application and reports that certain files have appeared in certain folders, and all this data is duplicated in Storj. To create such a synchronization system, we developed a rather complex background service that constantly monitors the presence of new folders or files in the selected folders of the Android user and synchronizes them.

For iOS, it was easier, because there you can only sync the Photos folder, so in iOS, if you enable sync, the system monitors your photos for updates and automatically syncs them.
We also implemented a camera function: you can take pictures in the application and immediately send the photo for storage in Storj. It is also possible to replenish your account – with bitcoins or Storj-coins –  by integrating such a bitcoin processor as BitPay (one of the world's largest bitcoin processors).

We also implemented a camera function: you can take pictures in the application and immediately send the photo for storage in Storj. It is also possible to replenish your account – with bitcoins or Storj-coins – by integrating such a bitcoin processor as BitPay (one of the world's largest bitcoin processors).

We also made a dashboard that displays the amount of data that the user downloads during the month, since billing is based on the pay as you go principle.

This was our first big case – we implemented it for Storj Labs in four to five months.

case-4
case-5
case-6

Mirroring Service application

Our second case was implemented quite quickly, in a few months – this is the Mirroring Service application. What is it, essentially? It was hypothesized that large companies that store data in the cloud do not want to move to a decentralized blockchain solution, fearing the loss of sensitive data, but they would not mind experimenting with blockchain technology.

For such companies, we developed the Mirroring Service, the idea of ​​which was as follows: the company sends a document for storage, for example, to Amazon Web Services, and also sends it to Storj Labs. To do this, we have made an application that "catches" the file at the moment it is sent, and distributes it to two places at the same time. At the end of the month, a company employee could compare the cost of storing files in two locations and see if Storj was cheaper and safer to store data. At the same time, this tool allowed Storj customers not to switch to Storj immediately, and thus not risking their data. It was a fairly quick project, which took two to three months.

Satellite Account

Then we started to implement one of our largest projects – the creation of the so-called Satellite Account. After Storj conducted the ICO, the company came to the conclusion that the storage system that worked before the ICO was very inefficient – it could not be scaled.

At a certain stage, Storj grew to 200 000 operator nodes, and the following problem was present: if you downloaded a file and wanted to delete it by clicking on the "delete" button, then the node on which this file was for some reason went offline and did not receive your command to delete, and the file remained on the node forever, since this file was no longer in the database, but it was saved on the node because the node was not online when the system notified it that the file needed to be deleted

As a result, since payments in Storj are still centralized – at the end of the month, the Storj team calculates how much disk space each node provided and how much data it processed, and based on this it determines how much the node should receive – then if the node has 10 GB, you need to pay for 10 GB. However, if 9 GB is removed and you were offline, then 10 GB is still left and you should pay for it. As a result, a huge amount of "garbage" data was generated in the system, which in the long term could lead to a collapse.

Accordingly, in March 2018, Storj completely reformatted the development team – our employees were included in the updated team. The company has a new CTO JT Olio, who is the CTO in one of the world's largest manufacturers of smart home devices – Vivint Smart Home. He proposed a completely new concept of creating a decentralized cloud-storage, the so-called enterprise-grade (corporate level), with which various business companies could work.

The idea was to create a not entirely decentralized cloud storage – rather, a "federated" one. It was understood that there is a network of node-operators, which is decentralized, and there are so-called "satellites": centralized applications with which users can create accounts in the usual interface, add their payment cards, add cryptocurrencies to replenish an account, create projects, and create API keys, using them to enter applications for loading data into a network of node-operators called Uplink – a command-line interface. 

You add an API key to Uplink and, thus, actually sign your files with this key. When uploading a file, you must add an API key, and when uploading a file, you must specify in Uplink which file you must upload and add an API key – this is the only way Uplink can find your files. Accordingly, without an API key, you cannot upload or download files.

Our task was to create a satellite account – in fact, a site where you could create an account to store your data, add payment methods, create an API key, create a project that will store data, plus visualization of projects – for example, data folders to visualize the account (payment for services, creating projects, creating keys, viewing statistics, generating accounting data, reports, invoices).
 

As for the payment part, we have integrated with such a serious payment provider as Stripe, and have implemented many use-cases for prepayments, post-payments, issuing invoices, charges, automatic charges, priority payment methods, non-priority payment methods, etc. Also, for payments, we integrated such a payment system as CoinPayments – at one time, it was one of the largest crypto payment companies.

What is the specialty of CoinPayments? You enter the request on their website "I want to top up my account with $100", in Storj tokens, and we automatically recalculated at the market rate how many people should send us Storj coins in order for him to top up his balance by $100, since everything balances – in dollars; the token performs more of a payment function, while all calculations are in dollars. The user pressed the button “pay 100 dollars with Storj tokens”, we redirected him to a special invoice page Coin Payments, where the wallet address was indicated, the wallet QR code and the transaction status were indicated. 

And you could transfer Storj-coins from your crypto-wallet to this specially generated crypto-wallet Storj-coins and so to top up your account. You replenished your account, after replenishment, the page was automatically destroyed, and $100 appeared on your account, after which you could pay for your services with them (payment was mainly made at the end of the month).

In addition, we have implemented a referral system and a bonus system: if you paid in Storj-coins, then for every $100 you were given $15 on top, and for the "invited friend" you were given a certain amount of free megabytes.

The above project was quite long and voluminous, since it implied integration with many different systems and technologies: with two payment systems, with different Storj databases, which "pull" completely different pieces of information – the number of megabytes that a particular account consumed per month, the number of megabytes that the account has uploaded and the number of uploaded objects. These three factors affect the price. It was a giant project that we have been doing for half of 2018 and throughout 2019.

In December 2019, Storj underwent a reorganization, and I was offered the position of a top manager at Storj, and also offered to split my team into several groups. So, starting in January 2020, our team worked in the format of three different teams, thanks to which we began to work three times more productively.
 

Storj Node Operator Dashboard

Our biggest project was the development of all kinds of products for operators of Storj nodes (users who rent disk space). The largest project of 2020 was the so-called SNO dashboard (Storj Node Operator Dashboard). Dashboard is an information panel that displays information about how much disk space of the operator nodes was used by the system, how much traffic was loaded, how much unloaded, what is the state of the disk at a particular moment – how free it is, how much "garbage" data is on it.

A notification system has also been implemented. Dashboard displays node-ID, by which you can check your data within the network, see node statistics (online or offline, what version it has); displays a crypto wallet with a link to Etherscan, by which you can easily view the statistics of your payments; a section with the user's reputation and a section of payments (pay out), containing information about the user's earnings, with all the historical data and a very detailed categorization.

In addition, in 2021, we improved the dashboard, adding a newfangled solution in the field of cryptocurrencies – the second layer protocol (Layer-2 protocol). There is a zkSync company whose solution allows you to send transactions to Ethereum wallets, but not within the Ethereum blockchain, but within the sidechain. We implemented this solution for Storj and thus reduced transaction costs – several tens of thousands of dollars per month to almost zero.
 

Dashboard

Users will be able to collaborate on project development and generate API keys.

Linux Installer and Updater

Our other project is Linux Installer and Updater, an application that allows you to deploy Linux nodes with one click, as well as update nodes automatically. In addition, we have improved the Windows installer, developed before us by another team, so that Linux and Windows users can now easily deploy nodes and share their disk space.
 

Wireframes & GUI Components

The Storj is AWS S3 compatible decentralized cloud object storage.
This is affordable, easy to use, private and secure place.

Color Scheme

color-scheme
inter-fonts