Video Tutorial#
Introduction and Project Features#
In today's environment with increasing file storage demands, I have successfully built a highly efficient and flexible private file storage cluster, combining the advantages of R2, GitHub, and GitLab across multiple platforms to form a highly reliable, distributed storage system. The core highlights of this solution include:
- Multi-node distributed storage: Files are stored according to a strategy, avoiding single points of failure.
- Automated backup and synchronization: Data is efficiently synchronized across different nodes, ensuring consistency and security.
- Intelligent access acceleration: With the help of Cloudflare's global edge services, access speed is greatly enhanced, providing a smooth user experience.
- Unlimited scalability potential: Thanks to the distributed architecture, it can theoretically store an almost unlimited amount of files.
This is not just a simple image hosting solution; it is a multifunctional distributed storage system suitable for code, documents, and any data storage.
Deployment Requirements#
- GitHub account, https://github.com
- GitLab account, https://gitlab.com
- Cloudflare account, https://www.cloudflare.com
- Install PicGo, https://molunerfinn.com/PicGo
Process and Tool Introduction#
PicGo Upload Files: Use PicGo and install the s3 plugin to upload files to Cloudflare R2 buckets.
GitHub action 1: Regularly migrate files from the Cloudflare R2 bucket to GitHub, loading storage files.
GitHub action 2: Regularly mirror GitHub nodes to GitLab to avoid the risk of losing all files.
Get links rewritten by Worker: Use Worker to hide the PAT of GitHub, GitLab, and R2 private repositories, outputting a custom domain URL.
Utilize global Edge CDN for accelerated access: Cache static files through Cache Rules to reduce origin requests, significantly improving access speed and stability.
Create n Private Repositories on GitHub as Nodes and Obtain a PAT#
Visit https://github.com/new, for example, create 3 private repositories with the same prefix and numerical suffix, such as node-1, node-2, node-3 ...
Create PATs for these 3 private repositories by visiting https://github.com/settings/tokens
Create n Identically Named Private Repositories on GitLab as Nodes, Obtain n PATs and Repo IDs#
Visit https://gitlab.com/projects/new#blank_project, create 3 identically named private repositories, ensuring they have the exact same names as the previously created repositories, with the same quantity.
Each of n CloudFlare Accounts Creates an R2 Bucket and Obtains n API Keys#
Record All PATs and Project IDs#
PicGo Compression Plugin s3#
After installing PicGo, continue to install the s3 plugin within the software.
Fill in the custom link format as https:///$fileName$extName
File path files/{fileName}.{extName}
Create a Private Repository on GitHub and Run Two Scheduled Tasks#
https://github.com/fscarmen2/pic-hosting-cluster to obtain the code, create a private repository on GitHub for scheduled synchronization tasks, and copy the code from my repository.
Official Usage Process#
Upload Files#
Upload files to GitHub via PicGo, using the shortcut (default Ctrl + Shift + P) or dragging files into the PicGo interface.
The upload progress bar is blue if successful; if it is red, please check if the settings are incorrect. After a successful upload, the custom domain link will automatically be in the clipboard.
Manually Run Scheduled Tasks in GitHub Action#
Advanced#
The parameters of the URL are as follows:
| Parameter query | Description |
|---|---|
| https:/// | Check the status of each node |
| https:///?from=r2 | Specify to get from R2 |
| https:///?from=github | Specify to get from GitHub |
| https:///?from=gitlab | Specify to get from GitLab |
| https:///?from=where | Query file and ownership node information |
Check the Status of Each Node#
Specify to Get File from R2#
Specify to Get File from GitHub#
Specify to Get File from GitLab#
Query File and Ownership Node Information#
Summary#
Through this solution, we have successfully built an efficient distributed file storage cluster that integrates R2, GitHub, and GitLab. This architecture not only enhances the security and backup reliability of files but also fully utilizes Cloudflare's global Edge CDN acceleration capabilities, making file access faster and more stable.
Whether storing images, code, or other important files, this cluster can meet the needs in a cost-effective and efficient manner. I hope this solution can inspire and facilitate your work and study. If you are interested in this cluster, why not try building it and enjoy the fun of innovation!