GitHub private repository image hosting cluster, PicGO upload, Cloudflare Workers acceleration, Gitlab real-time backup

Video Tutorial#

Introduction and Project Features#

Due to the extensive use of images in newly created blogs, a private image hosting platform needs to be established. It is necessary to avoid using paid commercial cloud OSS storage; find a long-term stable storage platform under the premise of free usage; ensure access speed; and ideally have off-site disaster recovery. To achieve these goals, the following comprehensive solution has been developed.

Solution Features and Advantages:

Image Hosting Cluster Matrix: A storage system composed of multiple GitHub repositories. It breaks through the 4GB size limit of each repository. Each repository acts as a node, forming an overall resource storage network. Different repositories are independently managed while collectively forming a distributed multi-repository cluster.
Image Storage in GitHub Private Repository: Images are stored in a GitHub private repository, with paths processed for desensitization to ensure security.
Automatic Image Compression: Automatically compressing images to improve loading speed and optimize user experience.
Cloudflare CDN Acceleration: Utilizing the global acceleration service provided by Cloudflare CDN to ensure fast image loading.
Custom Domain Name: Supports the use of a custom domain name, with multiple image libraries in the cluster unified under one domain name, enhancing brand image and professionalism.
Multiple Link Formats Output: Generates various forms of image links for easy use on blogs or other platforms.
Real-time Backup to GitLab Private Repository (Optional): Real-time backup from GitHub to GitLab private repository provides data redundancy and cross-platform backup, enhancing data security and ensuring business continuity.

Requirements#

GitHub Account, https://github.com
GitLab Account, optional; can be ignored if backup is not needed, https://gitlab.com
Cloudflare Account, https://www.cloudflare.com
Tinypng Account, https://tinypng.com/developers
Install Snipaste, https://zh.snipaste.com
Install PicGo, https://molunerfinn.com/PicGo

Process and Tool Introduction#

Screenshot: Use Snipaste for screenshots, a simple yet powerful screenshot tool that supports screen capture, annotation, and more, suitable for Windows and Mac.

PicGo Upload Images to GitHub: Use PicGo and install the picgo-plugin-compress-next plugin to automatically compress images before uploading them to GitHub.

Obtain Image Links Processed by Worker: Automatically generate links for images processed by Worker, hiding the GitHub private repository's PAT, increasing CDN and dual-stack, and outputting custom domain URLs.

Automatically Push Latest Images from GitHub to GitLab: Use GitHub Actions to automatically back up to GitLab whenever there is an update, avoiding the risk of losing all images.

Detailed Key Steps#

Prerequisites: Obtain GitHub PAT and Tinypng API Key#

Log in to GitHub, visit https://github.com/settings/tokens to obtain PAT, and create a private repository for storing images. Currently, each project on GitHub is 4GB in size; if exceeded, just create a new one.

Create a private repository in GitHub for storing images, visit https://github.com/new, and note that each project is 4GB in size; if exceeded, just create a new one.

PicGo Settings#

After installing PicGo, continue to install the picgo-plugin-compress-next plugin in the software (this step may require a VPN).
Continue to set up the PicGo main program.

Create Worker in Cloudflare#

Log in to Cloudflare, visit https://dash.cloudflare.com/, and create a new worker.
After deployment, edit the code, copying and modifying the corresponding two locations as shown in the image.

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})

async function handleRequest(request) {
  const GITHUB_USERNAME = '' // Set your GitHub username
  const GITHUB_PAT = ''  // Set your GitHub PAT (Personal Access Token)
  const GITHUB_REPO_PREFIX = 'pic'        // Repository prefix, needs to be used with REPO_COUNT
  const REPO_COUNT = 10                   // Number of repositories. For example, fill in 10, combined with the prefix to be pic1, pic2, ..., pic10
  const DIR = 'images'  // Directory path in the repository

  // Get the filename from the request URL
  const REPOS = Array.from({ length: REPO_COUNT }, (_, i) => `${GITHUB_REPO_PREFIX}${i + 1}`)
  const url = new URL(request.url)
  const FILE = url.pathname.split('/').pop()  // Get the last part of the URL as the filename

  // Build a list of URLs for GitHub raw files
  const urls = REPOS.map(repo => `https://raw.githubusercontent.com/${GITHUB_USERNAME}/${repo}/main/${DIR}/${FILE}`)

  // Create concurrent request tasks to fetch data from all GitHub raw file URLs
  const requests = urls.map(githubUrl => {
    const modifiedRequest = new Request(githubUrl, {
      method: request.method,
      headers: {
        'Authorization': `token ${GITHUB_PAT}`,  // Use GitHub PAT for authorization
        'Accept': 'application/vnd.github.v3.raw'
      }
    })
    return fetch(modifiedRequest).then(response => {
      if (response.ok) return response; // If the response is successful, return it
      throw new Error(`Not Found in ${githubUrl}`); // If the response is not successful, throw an error
    })
  })

  try {
    // Wait for the first successful request to return a result
    const response = await Promise.any(requests)

    // Create a new response, removing the Authorization header to avoid information leakage
    const newResponse = new Response(response.body, response)
    newResponse.headers.delete('Authorization')

    return newResponse

  } catch (error) {
    // If all requests fail, return a 404 error
    return new Response(`404: File Not Found (${FILE}) in any repository`, { status: 404 })
  }
}

Add a custom domain name; sometimes it may not take effect immediately, so you can try using the domain provided by the worker https://<workerName>.<cloudflareUser>.workers.dev first.

Official Usage Process#

Screenshot#

Use Snipaste to take screenshots to the clipboard; you can use the shortcut key (default Ctrl + F1).

Upload Images#

Upload images to GitHub via PicGo; you can use the shortcut key (default Ctrl + Shift + P) or drag and drop images into the PicGo interface.
The upload success progress bar is blue; if it is red, please check if the settings are incorrect. After a successful upload, the custom domain link will automatically be in the clipboard.
Paste it into the appropriate scene, such as a blog or forum.

Real-time Backup to GitLab (Optional)#

GitLab Settings#

Log in to GitLab, create a new Project, visit https://gitlab.com/projects/new#blank_project.
Add PAT and permissions.
Set to allow forced pushes so that the GitHub workflow can push the project over.

GitHub Settings#

Set up 3 secrets.

Name	secret
GITLAB_USERNAME	GitLab username
GITLAB_REPO	Project name created on GitLab
GITLAB_PAT	The PAT generated for the GitLab project above

Create a workflow to back up to GitLab, with the path and file as

.github/workflows/sync-to-gitlab.yml
The code is as follows:

name: Sync to GitLab

on:
  push:
    branches:
      - main
  workflow_dispatch:

jobs:
  sync:
    runs-on: ubuntu-latest

    env:
      GITLAB_USERNAME: ${{ secrets.GITLAB_USERNAME }}
      GITLAB_REPO: ${{ secrets.GITLAB_REPO }}
      GITLAB_PAT: ${{ secrets.GITLAB_PAT }}

    steps:
    - name: Checkout repository
      uses: actions/checkout@v4.1.1
      with:
        fetch-depth: 0

    - name: Set up Git
      run: |
        git config --global user.name 'github-actions'
        git config --global user.email 'github-actions@github.com'

    - name: Add GitLab remote
      run: git remote add gitlab https://${{ env.GITLAB_USERNAME }}:${{ env.GITLAB_PAT }}@gitlab.com/${{ env.GITLAB_USERNAME }}/${{ env.GITLAB_REPO }}.git

    - name: Force push to GitLab
      run: git push gitlab main --force

Image Example#

https://img.forvps.gq/pic1/main/example/202410021622722.png