Save System, Read & Write Any Data to Compressed Binary Files

From Epic Wiki

Custom Save System To Binary Files

Original Author: ( )

Dear Community,

Using what I explain in this tutorial you can write your own custom save systems wherein you

  • write out literally any game-related data you want
  • read back in this data any time you want, from hard-disk
  • compress (ZLIB) these files to minimize usage of the end-user's computer hard disk space

I've already been writing out and loading back in levels that are zipped binary data files for my in-game editor.


I am currently able to save literally any custom data I want, including dynamic arrays, for all my custom classes.


I've developed a streamlined method for doing all this by overloading a specific UE4 C++ operator, so yay thank you thank you Epic for C++ access!


Pre-Summary

There's a lot of concepts in this tutorial, try out each part and get it working before moving on to the next

Just copy-pasting code will not work here, you need to understand the basics, because all I am presenting is the basics,

and you must apply the basics to your own project's needs.

After all, this is about a save system for any arbitrary project-specific custom class data :)


Two Levels of Conversions

When you want to save custom variable data for your custom save system,

there are TWO major steps involved

Step 1 = variable format -> binary array (serialized by Archive class of UE4)

Step 2 = Binary array -> hard disk

These steps are then done in reverse to read back in data from hard disk.


Binary Array = TArray<uint8>

Binary data is represented in a very UE4 C++ friendly looking way as a dynamic array of uint8.

So any time you see TArray<uint8> in my code in this tutorial, that literally means "Binary Array" from UE4 C++ standpoint.


Step 1: Variable Format -> Binary Array

An int32 takes up 4 bytes, as does a float.

An int64 takes up 8 bytes.

An FName takes up 8 bytes.

An FString takes up 16 bytes.

An FVector takes up 3 x float bytes.

etc.

So even a single int32 is actually an array of bytes, not a single entry.

Now let's suppose your save system needs to store

  • 3 FVector's
  • 40 int32's
  • 20 FName's

Do all the math and this is a lot of bytes!

3 x 3 x 4 = 36

40 x 4 = 160

20 x 8 = 160

356

So this means that before your data leaves UE4 and goes to hard disk, you need a TArray<uint8> that has 356 entries.

Step 2: Binary Array -> Hard Disk

UE4 C++ gives you functions via FileManager.h to write out TArray<uint8> to hard disk!


Optional Step 3: Compressed Binary

UE4 gives you functionality via Archive.h to compress a TArray<uint8> before sending it to the FileManager

C++ Code For You

Below I am giving you the functions that I use to to read and write binary files of any custom data I choose


Core Header Files

Archive.h and ArchiveBase.h

See Archive.h and ArchiveBase.h for all the info you need about getting from your varibles and custom class data to binary format (serialized data).

FileManager.h

All the functions you need to

  • create directories
  • delete directories
  • create files
  • delete files
  • get a listing of all files in a given path
  • get a listing of all folders in a given path
  • get the age of a file

and more are found in FileManager.h

You access these functions from anywhere using 

if(GFileManager) GFileManager->TheFunction()

BufferArchive

The buffer archive is both a binary array (TArray<uint8>), and a MemoryWriter

Archive.h 

/**
 * Buffer archiver.
 */
class FBufferArchive : public FMemoryWriter, public TArray<uint8>
{

Because of this multiple-inheritance, the BufferArchive is my preferred way to write data to binary file.

As my code will show, because the GFileManager wants to receive a TArray<uint8>, not a MemoryArchive.

Review my Steps 1 and 2 to see why this is such an awesome class. :)

Thanks UE4 Devs!


FMemoryReader

To read Data back from a binary array, that is retrieved by the FileManager, you need a MemoryReader

Archive.h 

/**
 * Archive for reading arbitrary data from the specified memory location
 */
class FMemoryReader : public FMemoryArchive
{
public:

The << Operator

The BufferArchive/Binary Array needs to retrieve your game's variable data, how do you tell it what you want stored as binary?

The << operator!


Variable -> Binary

Here's how you would put an FVector into a BufferArchive to then be saved to hard disk.

//in player controller class 

FBufferArchive ToBinary;
ToBinary << GetPawn()->GetActorLocation(); //save player location to hard disk

//save ToBinary to hard disk using File Manager, 
//see complete code samples below

Binary -> Variable

Here's how you would retrieve an FVector from a TArray<uint8> as retrieved by GFileManager. 

//TheBinaryArray was already obtained from FileManager, 
//see code below for full examples

//need to supply a variable to be filled with the data
FVector ToBeFilledWithData;

FMemoryReader Ar = FMemoryReader(TheBinaryArray, true); //true, free data after done
Ar.Seek(0); //make sure we are at the beginning

Ar << ToBeFilledWithData;

The Hardest Concept of UE4 C++ Custom Save System

Compare these two lines 

ToBinary << GetPawn()->GetActorLocation();
Ar << ToBeFilledWithData;

The hardest concept for me about the UE4 archive system was the fact that the << operator could mean

- getting data out of the archive and putting it into the variable

or

- putting data from the variable into the archived binary format

depending on the context!


So that's why I recommend you name your BufferArchive something like ToBinary,

and your MemoryReader something totally different ,

you can only discern the difference between writing to binary and reading from binary

based on the context as you show it in your code, as the << operator will tell you nothing from a simple glance.


Writing Your Function to Be Two-Way

The critical advantage of this system though is that you can write a single function that works both ways.

So you can write a single function that loads data from file, or saves to file.

But why would you want this?

Because:

The order of how you write out data to binary file must be the exact order that you read it back in!

The computer does not have any way of knowing, nor does UE4, what the correct order of variable data should be.

You are responsible for telling the computer and UE4 to read data back in in the same order it was written out to file.

Thus, having a single function that both reads and writes, using the multi-purpose << operator, is the safest thing you can do to ensure consistency of writing/reading binary data.

SaveLoadData: Two-Way Save System Function 

'''.h'''

//FArchive is shared base class for FBufferArchive and FMemoryReader
void SaveLoadData(FArchive& Ar, int32& SaveDataInt32, FVector& SaveDataVector, TArray<FRotator>& SaveDataRotatorArray);

'''.cpp'''

//I am using controller class for convenience, use any class you want

//SaveLoadData
void YourControllerClass::SaveLoadData(FArchive& Ar,
  int32& SaveDataInt32,
  FVector& SaveDataVector,
  TArray<FRotator>& SaveDataRotatorArray
)
{
	Ar << SaveDataInt32;
	Ar << SaveDataVector;
	Ar << SaveDataRotatorArray;
}

Saving

Make a BufferArchive and pass it in, it is a Binary Array and also an FArchive 

FBufferArchive ToBinary;
SaveLoadData(ToBinary,NumGemsCollected,PlayerLocation,ArrayOfRotationsOfTheStars);
//save the binary array / FBufferArchive to hard disk, see below

Loading 

// TheBinaryArray already retrieved from file, see full code sample
FMemoryReader FromBinary = FMemoryReader(TheBinaryArray, true); //true, free data after done
FromBinary.Seek(0);
SaveLoadData(FromBinary,NumGemsCollected,PlayerLocation,ArrayOfRotationsOfTheStars);

Summary

Use this setup to avoid crashes due to reading data not in same order that you wrote it to disk!

this two way functionality of UE4 << operator saves the day!

Thanks Epic Devs!

My Binary Save System Functions For You

Below I am giving you the functions that I use to save/load binary files!

Saving Binary Files 

bool ControllerClass::SaveGameDataToFile(const FString& FullFilePath, FBufferArchive& ToBinary)
{
	//note that the supplied FString must be the entire Filepath
	// 	if writing it out yourself in C++ make sure to use the \\
	// 	for example:
	
	// 	FString SavePath = "C:\\MyProject\\MySaveDir\\mysavefile.save";
	
	//Step 1: Variable Data -> Binary
	
	//following along from above examples
	SaveLoadData(ToBinary,NumGemsCollected,PlayerLocation,ArrayOfRotationsOfTheStars); 
	//presumed to be global var data, 
	//could pass in the data too if you preferred

	//No Data
	if(ToBinary.Num() <= 0) return false;
	//~
	
	//Step 2: Binary to Hard Disk
	if (FFileHelper::SaveArrayToFile(ToBinary, * FullFilePath)) 
	{
		// Free Binary Array 	
		ToBinary.FlushCache();
		ToBinary.Empty();
		
		ClientMessage("Save Success!");
		return true;
	}
	
	// Free Binary Array 	
	ToBinary.FlushCache();
	ToBinary.Empty();
	
	ClientMessage("File Could Not Be Saved!");
	
	return false;
}

FBufferArchive

Loading Binary Files 

//I am using the sample save data from above examples as the data being loaded
bool ControllerClass::LoadGameDataFromFile(
	const FString& FullFilePath, 
	int32& SaveDataInt32,
	FVector& SaveDataVector,
	TArray<FRotator>& SaveDataRotatorArray
){
	//Load the data array,
	// 	you do not need to pre-initialize this array,
	//		UE4 C++ is awesome and fills it 
	//		with whatever contents of file are, 
	//		and however many bytes that is
	TArray<uint8> TheBinaryArray;
	if (!FFileHelper::LoadFileToArray(TheBinaryArray, *FullFilePath))
	{
		ClientMessage("FFILEHELPER:>> Invalid File");
		return false;
		//~~
	}

	//Testing
	ClientMessage("Loaded File Size");
	ClientMessage(FString::FromInt(TheBinaryArray.Num()));
	
	//File Load Error
	if(TheBinaryArray.Num() <= 0) return false;
	
	//~
	//		  Read the Data Retrieved by GFileManager
	//~
	
	FMemoryReader FromBinary = FMemoryReader(TheBinaryArray, true); //true, free data after done
	FromBinary.Seek(0);
	SaveLoadData(FromBinary,NumGemsCollected,PlayerLocation,ArrayOfRotationsOfTheStars);

	//~
	//								Clean up 
	//~
	FromBinary.FlushCache();
	
	// Empty & Close Buffer 
	TheBinaryArray.Empty();
	FromBinary.Close();
	
	return true;
}

Saving Compressed 

bool ControllerClass::SaveGameDataToFileCompressed(const FString& FullFilePath, 
	int32& SaveDataInt32,
	FVector& SaveDataVector,
	TArray<FRotator>& SaveDataRotatorArray
){
	FBufferArchive ToBinary;
	SaveLoadData(ToBinary,NumGemsCollected,PlayerLocation,ArrayOfRotationsOfTheStars); 
	
	//Pre Compressed Size
	ClientMessage("~ PreCompressed Size ~");
	ClientMessage(FString::FromInt(ToBinary.Num()));
	
	//
	
	// Compress File 
	//tmp compressed data array
	TArray<uint8> CompressedData;
	FArchiveSaveCompressedProxy Compressor = 
		FArchiveSaveCompressedProxy(CompressedData, ECompressionFlags::COMPRESS_ZLIB);
	
	//Send entire binary array/archive to compressor
	Compressor << ToBinary;
	
	//send archive serialized data to binary array
	Compressor.Flush();
	
	//
	
	//Compressed Size
	ClientMessage("~ Compressed Size ~");
	ClientMessage(FString::FromInt(CompressedData.Num()));
	
	
	if (!GFileManager) return false;
	
	//vibes to file, return successful or not
	if (FFileHelper::SaveArrayToFile(CompressedData, * FullFilePath)) 
	{
		// Free Binary Arrays 
		Compressor.FlushCache();
		CompressedData.Empty();
		
		ToBinary.FlushCache();
		ToBinary.Empty();
		
		// Close Buffer 
		ToBinary.Close();
		
		ClientMessage("File Save Success!");
		
		return true;
		//
	}
	else
	{
		// Free Binary Arrays 
                Compressor.FlushCache();
		CompressedData.Empty();

		ToBinary.FlushCache();
		ToBinary.Empty();
		
		// Close Buffer 
		ToBinary.Close();
		
		ClientMessage("File Could Not Be Saved!");
		
		return false;
		//
	}
}


Loading Compressed 

//I am using the sample save data from above examples as the data being loaded
bool ControllerClass::LoadGameDataFromFileCompressed(
	const FString& FullFilePath, 
	int32& SaveDataInt32,
	FVector& SaveDataVector,
	TArray<FRotator>& SaveDataRotatorArray
){
	//Load the Compressed data array
	TArray<uint8> CompressedData;
	if (!FFileHelper::LoadFileToArray(CompressedData, *FullFilePath))
	{
		Optimize("FFILEHELPER:>> Invalid File");
		return false;
		//~~
	}
	
	// Decompress File 
	FArchiveLoadCompressedProxy Decompressor = 
		FArchiveLoadCompressedProxy(CompressedData, ECompressionFlags::COMPRESS_ZLIB);
	
	//Decompression Error?
	if(Decompressor.GetError())
	{
		Optimize("FArchiveLoadCompressedProxy>> ERROR : File Was Not Compressed ");
		return false;
		//
	}
	
	//Decompress
	FBufferArchive DecompressedBinaryArray;
	Decompressor << DecompressedBinaryArray;
	
	//~
	//		  Read the Data Retrieved by GFileManager
	//~
	
	FMemoryReader FromBinary = FMemoryReader(DecompressedBinaryArray, true); //true, free data after done
	FromBinary.Seek(0);
	SaveLoadData(FromBinary,NumGemsCollected,PlayerLocation,ArrayOfRotationsOfTheStars);

	//~
	//								Clean up 
	//~
	CompressedData.Empty();
	Decompressor.FlushCache();
	FromBinary.FlushCache();
	
	// Empty & Close Buffer 
	DecompressedBinaryArray.Empty();
	DecompressedBinaryArray.Close();
	
	return true;
}


Overloading the << Operator

Create your own << Operator overloads to simplify the process!

Let's say you have your own USTRUCT or your own class, and you want to write a way to

simply write 

ToBinary << MyEntireSaveSystem;

or 

ToBinary << MySpecialUStruct;


Here's how you overload the << Operator!

Please note this must be in the .h file, and no contents in the .cpp.

Also there is no context, such as MyClass::, it must be at the global level.

Also the .h file that has this definition must be compiled before any classes that want to use it.

You can use UClass(dependson=UYourDefinitionsClass) to ensure this

or simply put the .h contents in your public directory

and include them as a header somewhere.

.h 

//Make as many Unique Overloads as you want!
FORCEINLINE FArchive& operator<<(FArchive &Ar, UMySaveGameClass* SaveGameData )
{
	if(!SaveGameData) return Ar;
	//~
	
	Ar << SaveGameData->NumGemsCollected;  //int32
	Ar << SaveGameData->PlayerLocation;  //FVector
	Ar << SaveGameData->ArrayOfRotationsOfTheStars; //TArray<FRotator>

        return Ar;
}

Note:

1. The operator is returning the Ar by reference.

2. No const are allowed at global level (its a compile error)

3. No const allowed inside because you dont know if you are reading or writing due to nature of << operator.

Crashes?

If you get crashes you are not reading/writing data in same order,

use overloaded << operator and the idea of a single function like SaveLoadGame to avoid this


Also, if you compress data before saving it make sure you are loading it using my compressed function , not the regular one, and vice versa :)

Enjoy!

Have fun making your very own custom save game system, and saving it to compressed binary file!

♥

( )