fscarmen

fscarmen

HomeArchives

First release on the entire network! Say goodbye to traditional single-node image hosting! GitHub and GitLab join forces to form a cluster, opening a new chapter in image storage.

#cdn#worker#github#picgo#workflows#gitlab54
AI Translation
This post is translated from Chinese into English through AI.View Original
AI-generated summary
The document outlines a revolutionary image hosting solution based on a cluster architecture using GitHub and GitLab. It addresses the limitations of traditional single-node storage systems, which struggle to meet growing storage demands and access speed requirements. The proposed solution offers several advantages: 1. **Unlimited Storage**: By utilizing multiple nodes, it overcomes capacity limitations, allowing for the storage of vast amounts of images without constraints. 2. **Data Security**: Automatic synchronization across nodes ensures data safety, with backups on both GitHub and GitLab. 3. **Smart Monitoring**: Users can easily check the status of each node with a custom password, enhancing management and stability. 4. **Unified Access**: A single custom domain simplifies access to the image hosting service. 5. **Concurrent Access**: The system selects the fastest node for data transfer, optimizing access speed. 6. **Cloudflare CDN**: Integration with Cloudflare enhances global access speed. 7. **Image Optimization**: Uploaded images are compressed to save space and improve loading times. The document also details the necessary accounts, tools, and steps for setting up the system, including creating repositories, generating tokens, and configuring the PicGo tool for image uploads. It emphasizes the importance of automated data synchronization to prevent data loss.

Video Tutorial#

Introduction and Project Features#

In today's digital wave, images have become an indispensable key element in information transmission. Whether it's the exquisite illustrations that embellish personal blogs or the rich materials that support various applications, the effectiveness, security, stability, speed, and cost-effectiveness of their storage and management are the core goals that current solutions are competing to achieve. Unfortunately, most mainstream solutions still remain limited to traditional single-node storage models, which have proven inadequate in the face of ever-expanding storage demands and increasing access speed requirements.

Against this backdrop, I have designed a revolutionary solution—a cluster architecture for image hosting based on GitHub and GitLab. This architecture aims to completely break traditional constraints and lead personal image storage management to new heights. By introducing a k8s-like clustering mindset, we hope to provide a comprehensive image hosting solution that meets massive storage needs, ensures access speed, and balances cost-effectiveness.

Advantages of the Solution:

  • Break storage limits, unleash infinite possibilities: Using a cluster architecture, multiple nodes working in coordination through GitHub and GitLab completely break the capacity bottleneck of a single repository. Whether it's massive images or high-definition pictures, it can easily handle them, ensuring that the capacity of storage objects is no longer a shackle on your creativity.

  • Data security without worries, multiple backups safeguard: Data security is one of our top priorities. This solution adopts a strategy of automatic synchronization of data across multiple nodes, ensuring the safety and reliability of the data. Additionally, data can not only be stored on a single node of GitHub but is also mirrored on GitLab, forming a dual guarantee. Even if one platform encounters any issues, the data remains intact and worry-free.

  • Intelligent monitoring, easily grasp node status: In this solution, you can easily obtain the working status of each node in the cluster gallery with a custom password. This innovative design not only simplifies management processes but also enhances system maintainability and stability.

  • Unified domain name, simplified access experience: Although the image hosting cluster of this solution consists of multiple nodes, users only need a unified custom domain name for easy access. This design not only simplifies the usage process but also ensures uniformity and consistency in access, making your project more professional and reliable.

  • Concurrent access, speed is king: In terms of access speed, this solution also spares no effort. By concurrently accessing GitHub and GitLab, it automatically selects the node with the fastest response speed for data transmission. This innovative design ensures that users can obtain the required content at the fastest speed, enhancing user experience and satisfaction.

  • Cloudflare CDN support, global acceleration: This solution specifically introduces Cloudflare CDN, leveraging its global edge computing capabilities to significantly enhance access speed. No matter where users are located, they can enjoy a silky-smooth access experience.

  • Image compression optimization, improve loading speed: To further enhance user experience, this solution compresses all uploaded images. By reducing file sizes, it not only saves storage space but also significantly improves webpage loading speed.

Requirements#

Process and Tool Introduction#

Screenshot: Use Snipaste for screenshots, a simple yet powerful tool that supports screen capture, annotation, and more, suitable for Windows and Mac.

PicGo Upload Images to GitHub: Use PicGo and install the picgo-plugin-compress-next plugin to automatically compress images before uploading them to GitHub.

Obtain image links processed by Worker: Automatically generate a PAT that hides GitHub and GitLab private repositories through Worker, using the personal CDN and dual-stack to output custom domain URLs.

Scheduled automatic data synchronization: Use GitHub actions or GitLab pipeline scheduled tasks to regularly back up data from GitHub and GitLab nodes, avoiding the risk of losing all images.

Detailed Key Steps#

Create n private repositories on GitHub as nodes, obtain a PAT#

Visit https://github.com/new, for example, create 3 private repositories with the same prefix and numerical suffix, such as node-1, node-2, node-3 ...

image

image

Create a PAT for these 3 private repositories by visiting https://github.com/settings/tokens

image
image
image
image

Create n identically named private repositories on GitLab as nodes, obtain n PATs and Repo IDs#

Visit https://gitlab.com/projects/new#blank_project, create 3 identically named private repositories that match the names of the previously created repositories.

image
image
image
image
image
image

Record all PATs and Project IDs#

image

PicGo Compression Plugin compress-next#

After installing PicGo, continue to install the picgo-plugin-compress-next plugin (this step may require scientific access).

image
image
image

PicGo Settings#

After installing PicGo, continue to install the picgo-plugin-compress-next plugin (this step may require scientific access).
image
image

Continue to set up the PicGO main program.

image
image
image

Configuration for the 3 GitHub nodes.

image
image
image

Create a worker on Cloudflare#

Log in to Cloudflare, visit https://dash.cloudflare.com/, and create a new worker.

image

After deployment, edit the code, copy the code from GitHub https://github.com/fscarmen2/pic-hosting-cluster and modify the corresponding positions as shown in the image.

image

  • Add a custom domain name; sometimes it may not take effect immediately, so you can try using the domain provided by the worker https://<workerName>.<cloudflareUser>.workers.dev.

image
image

Official Usage Process#

Screenshot#

  • Use Snipaste to take screenshots to the clipboard, you can use the shortcut key (default Ctrl + F1).

Upload Images#

Upload images to GitHub via PicGo, you can use the shortcut key (default Ctrl + Shift + P) or drag and drop images into the PicGo interface.

The progress bar is blue when the upload is successful; if it is red, please check if the settings are incorrect. After a successful upload, the custom domain link will automatically be in the clipboard, ready to be pasted into the relevant scenarios like blogs or forums.

image
image

Scheduled Synchronization to Corresponding Nodes on GitLab#

The scheduled synchronization task can be set on either GitHub or GitLab; the following explains both.

Create a scheduled task in GitHub Action#

Obtain the code from https://github.com/fscarmen2/pic-hosting-cluster, find the github_to_gitlab module, create a private repository on GitHub for the scheduled synchronization task, and copy the code from my repository.

image
image
image
image
image
image

Create a scheduled task in GitLab pipeline#

Obtain the code from https://github.com/fscarmen2/pic-hosting-cluster, find the github_to_gitlab module, create a private repository on GitLab for the scheduled synchronization task, and copy the code from my repository.

image
image
image
image
image
image

Advanced#

Check the status of each node#

https:///

image

Specify to get images from GitHub#

https:///?from=github

image

Specify to get images from GitLab#

https:///?from=gitlab

image

Query image and belonging node information#

https:///?from=where

image

Conclusion#

This solution, based on the GitHub and GitLab image hosting cluster architecture, brings a revolutionary change to image storage management with its groundbreaking innovative concept and outstanding performance. For individual users, this means you will have a more convenient and efficient way to manage your image resources, whether for personal blogs, social media sharing, or material management in game development, all of which can be greatly enhanced and simplified.

Loading...
Ownership of this post data is guaranteed by blockchain and smart contracts to the creator alone.