Troubleshooting Guide: Git LFS

Use this guide to resolve common problems you might encounter when cloning, fetching, pulling, or pushing large files tracked by Git LFS to AIOZ AI.

Best Practice: Track Large Files Before You Commit

This is the most important step to prevent problems. You should always tell LFS to track new types of large files (e.g., *.safetensors, *.csv) before you commit them.

Why? Committing large files directly to Git history (without LFS) makes the repository slow for everyone. It is also very difficult to fix later.

How to do it: Before you git add your file, run git lfs track.

For a new file type (e.g., all PyTorch models):
```
git lfs track "*.pth"
```
For a specific large file:
```
git lfs track "my-big-dataset.csv"
```

This will create/update a .gitattributes file. You should add and commit this file along with your new large file.

git add .gitattributes
git add my-big-dataset.csv
git commit -m "feat: Add new dataset, correctly tracked with LFS"
git push

1. Pushing/Fetching Fails with Timeouts

Your git push or git pull fails with errors like i/o timeout, EOF, or Connection timed out, especially on a slow or unstable connection.

Cause: The connection is dropping before the large file transfer can complete.

Solution:

Increase the LFS activity timeout. This gives LFS more time to complete the transfer without the connection being considered "dead."
Run this command in your terminal to set the timeout to 120 seconds (2 minutes):
```
git config --global lfs.activitytimeout 120
```
Note: The default is often 30 seconds. You can set this value even higher (e.g., 300 for 5 minutes) if you are on a very poor connection.

2. Error: File Size Exceeds Maximum Limit (8 GB)

You see an error message like:

[...] Size must be less than or equal to 8589934592: [422] Size must be less than or equal to 8589934592

Cause: You are trying to push a single file that is larger than the repository's configured limit.

Solution / Policy:

The maximum allowed size for a single file tracked by LFS in this project is 8 GB.
You must:
- Check the file: Verify it's not a mistake (e.g., an accidental export or a log file).
- Split the file: If the file is a legitimate artifact (like a model checkpoint or dataset), you must break it into smaller parts, each under 8 GB.
- Contact Support: If splitting the file is impossible, please email [email protected] to discuss a potential exception.

3. Large Files Appear as Small Pointer Files

After cloning or pulling, your large files (e.g., model.pth) are just small text files (around 130 bytes) that look like this:

version https://git-lfs.github.com/spec/v1
oid sha256:4d7a2146...
size 123456789

Cause: This means Git LFS isn't installed on your local machine, or it wasn't properly initialized before you cloned.

Solution:

Install Git LFS: Follow the official installation instructions for your operating system at https://git-lfs.com/ (opens in a new tab) (e.g., brew install git-lfs or apt-get install git-lfs).
Install LFS hooks: After installing, run this once per machine:
```
git lfs install
```
Pull the actual files: Now, fetch the large files from the LFS store to replace the pointers:
```
git lfs pull
```

4. Still Stuck?

If your problem isn't listed, or the solutions don't work:

Get Debug Info:

Re-run your failing command with GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1 at the beginning. This provides verbose logs.

Example: GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1 git push
Email us for support:

Please send an email to [email protected] with the following template:

Subject: Git LFS Troubleshooting Request - [Your Repo Name]

Body:

Hi Support,

I'm running into an issue with Git LFS and the troubleshooting guide didn't resolve it.

Summary of the Problem:

(Briefly describe what you were trying to do, e.g., "I'm trying to push a new model file, but it's failing.")

The Command I Ran:

(Paste the exact command, e.g., git push)

The Full Error Message:

(Paste the complete error text you see in your terminal)

Debug Output:

(Re-run your failing command with GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1 at the beginning and paste the full output.)