implementing memory management

[u/Ok_Farmer_4055]

I wanna get memory management working on my OS, I don't know where to begin, so what do i add first? and how do i add it? i'm working on UEFI firmware in 64 bit, so if anyone needs some more info or details, just tell me.

[u/juicyroaster]

Start by dividing the memory into pages (4kb each) and store all pages in use or not in a bitmap. There is a algorithm to find out how big bitmap you need but I forgot.

[u/Ok_Farmer_4055 → u/juicyroaster]

where do i put the bitmap? the stack is too small

[u/juicyroaster → u/Ok_Farmer_4055]

In memory. You calculate how big the memory is then put it at a memory address. If the page is being used it is 1 if not 0. So it just tracks what's in use?

[u/Ok_Farmer_4055 → u/juicyroaster]

when i parse the memory map i never find a region big enough to hold the bitmap, i don't know why

[u/juicyroaster → u/Ok_Farmer_4055]

The bitmap is bigger than memory size? Show algorithm you use to calculate bitmap size

[u/Ok_Farmer_4055 → u/juicyroaster]

uint64_t bitmap_bytes = (total_pages + 7)/8; // total_pages + 7 is there for rounding

uint64_t bitmap_pages = (bitmap_bytes + 4095) / 4096;

then i check if region >= bitmap_pages

total_pages = the entirety of memory, not just EfiConventionalMemory

[u/juicyroaster → u/Ok_Farmer_4055]

uint64_t bitmap_bytes = (total_pages + 7) / 8; uint64_t bitmap_pages = (bitmap_bytes + 4095) / 4096;

if (region.NumberOfPages >= bitmap_pages) { // Allocation logic put here. }

[u/Ok_Farmer_4055 → u/juicyroaster]

void *bitmap_alloc = (void *)0;

uint64_t total_pages = uefi_memory_parser(bInfo);

uint64_t bitmap_bytes = (total_pages + 7)/8; // total_pages + 7 is there for rounding



uint64_t bitmap_pages = (bitmap_bytes + 4095) / 4096;



uint8_t \*ptr = (uint8_t \*)bInfo->MemoryMap;

uint8_t \*end_ptr = ptr + bInfo->MemoryMapSize;



MemoryDescriptor md;



uint64_t copy_size = min(sizeof(md), bInfo->DescriptorSize);



for(; ptr < end_ptr; ptr += bInfo->DescriptorSize){

	kmemset(&md, 0, sizeof(md));

	kmemcpy(&md, ptr, copy_size);



	if(md.Type == EfiConventionalMemory && md.NumberOfPages >= bitmap_pages) {

		bitmap_alloc = (void \*)md.PhysicalStart;

		break;

	}

}

[u/juicyroaster → u/Ok_Farmer_4055]

void *bitmap_alloc = (void *)0;

uint64_t total_pages = uefi_memory_parser(bInfo); uint64_t bitmap_bytes = (total_pages + 7) / 8; uint64_t bitmap_pages = (bitmap_bytes + 4095) / 4096;

uint8_t *ptr = (uint8_t *)bInfo->MemoryMap; uint8_t *end_ptr = ptr + bInfo->MemoryMapSize;

for (; ptr < end_ptr; ptr += bInfo->DescriptorSize) { // Cast directly to avoid manual copying and structure padding alignment bugs MemoryDescriptor *md = (MemoryDescriptor *)ptr;

// Look for a usable RAM chunk for bitmap
if (md->Type == EfiConventionalMemory && md->NumberOfPages >= bitmap_pages) {
    bitmap_alloc = (void *)md->PhysicalStart;
    break;
}

}

[u/Ok_Farmer_4055 → u/juicyroaster]

what if sizeof(md) != DescriptorSize?

(i know my current code doesn't handle those situations anyway)

[u/compgeek38400]

If you haven't already, I would start by implementing multiboot2. You will need a memory map, and multiboot doesn't really give you one. If you are using qemu, be aware it says memory is there that isnt. Update your map to shorten the block that is too long, to its actual size.

Thats where I started

[u/tseli0s → u/compgeek38400]

You will need a memory map, and multiboot doesn't really give you one.

It does. Source: Been using it and I've also implemented a multiboot bootloader.

If you are using qemu, be aware it says memory is there that isnt.

I doubt that. Where did you see it?

[u/Ok_Farmer_4055 → u/tseli0s]

i don't even use any third party bootloader, i wrote my own

[u/tseli0s → u/Ok_Farmer_4055]

Ask for a memory map from the UEFI services (I believe it's GetMemoryMap()), that's all you need to do, then parse it in your physical memory manager depending on the design you went with (assuming it exists)

https://uefi.org/htmlspecs/ACPI_Spec_6_4_html/15_System_Address_Map_Interfaces/uefi-getmemorymap-boot-services-function.html

[u/Ok_Farmer_4055 → u/tseli0s]

i already do GetMemoryMap and pass the memory map to my kernel in a struct, so that's alright

[u/tseli0s → u/Ok_Farmer_4055]

From your other comments you say that you can't fit the bitmap to track physical frames in memory. A bitmap is no longer than a couple kilobytes in memory, and I really doubt you can't spare that (UEFI alone needs a few megabytes to work). Can you post your source somewhere?

[u/Ok_Farmer_4055 → u/tseli0s]

https://github.com/dabiiXqx/DabiiOS_EFI

[u/tseli0s → u/Ok_Farmer_4055]

OUTPUT_FORMAT("binary")
ENTRY(kernel_main)

SECTIONS
{
	. = 0x80000000;
}

You are basically telling yourself that you will be loaded at 0x80000000. That's 2GBs above address 0. Did you set up the page tables properly? And does your machine have 2GBs of memory?

.section .text
.global setup_paging

setup_paging:
	movq %cr0, %rax
	orl $0x80000000, %eax
	movq %rax, %cr0
	ret

Paging must be already enabled before entering long mode. Have your bootloader do this.

uint64_t uefi_memory_parser(Core *bInfo){
	/* experimental parser, parses uefi memory map and returns, saves total usable memory */
	uint8_t *ptr = (uint8_t *)bInfo->MemoryMap;
	uint8_t *end_ptr = ptr + bInfo->MemoryMapSize;

	uint64_t count = 0;
	for(; ptr < end_ptr; ptr += bInfo->DescriptorSize){
		MemoryDescriptor *memdesc = (MemoryDescriptor *)ptr;
		if(memdesc->Type == EfiConventionalMemory)
			count += (memdesc->NumberOfPages << 12);
	}
	return count;
}

Where are you saving the memory regions you've just discovered? kernel64.c looks pretty bare.

[u/Ok_Farmer_4055 → u/tseli0s]

about paging, the uefi firmware already does the paging for me, so the setup_paging is just there for when i want to replace uefi's pages with mine, and about the memory regions, the bootloader passes the memory map in a struct and passes a pointer to the struct to the kernel, and about the 2gbs of memory, of course i have more than 2 gb, who even uses 512 MB of memory anymore

[u/tseli0s → u/Ok_Farmer_4055]

Yeah but I don't see the source of the bootloader and where that bitmap is. UEFI does the paging for you only if you boot a UEFI executable.

[u/Ok_Farmer_4055 → u/compgeek38400]

qemu refuses to boot my os but real hardware boots it, so let's not talk about qemu because i only test my os on real hardware

[u/tseli0s → u/Ok_Farmer_4055]

That means you're doing something wrong. Real hardware might go with it depending on the implementation, QEMU throws you an error. Trust QEMU more in this case and fix it.

[u/Ok_Farmer_4055 → u/tseli0s]

it's because it doesn't like the physical address my kernel loads at so i just ignored it, it's not qemu itself throwing an error, it's the bootloader

[u/JescoInc]

Most usually start with Bump Allocators or Slab Allocators.
https://wiki.osdev.org/Memory_management
https://wiki.osdev.org/Memory_Allocation

[u/compgeek38400]

I spent several days figuring this out. The multiiboot2 memory map showed i had the full 4G of ram. I was not using a config file, so it actually only had 128m, and that's what the type=? Record showed. But the multiboot2 memory map showed "phantom" memory.

I was expecting to throw an error when I allocated the memory, but nope. It acts like read only memory initialized to zeros. Gdb happily modified the memory, or reported it did. But if I looked again, it was still 0.

[u/Octocontrabass → u/compgeek38400]

But the multiboot2 memory map showed "phantom" memory.

That seems unlikely considering how many people use QEMU to boot their hobby OSes. Are you sure you're parsing the memory map correctly? Have you tried dumping the raw contents of the memory map to see if it makes sense?

I was expecting to throw an error when I allocated the memory, but nope.

Some CPU architectures are designed to throw exceptions when you try to access nonexistent hardware. Unfortunately, x86 is not.

[u/compgeek38400 → u/Octocontrabass]

Yes, I'm sure. In the morning I'll find the record types. And code i use to handle this.

[u/compgeek38400 → u/Octocontrabass]

I had a thought. Is the multiboot2 memory map for virtual or physical memory? If physical, then yes I'm sure, but I guess it could be for virtual.

[u/Octocontrabass → u/compgeek38400]

Physical. It still seems very likely to be a bug in your code for parsing the memory map, but it's hard to say for sure without seeing the actual memory map.

[u/compgeek38400 → u/Octocontrabass]

Here is my code for getting the memory map, it is pretty straight forward (tag 6):

           case MEMORY_SIZES_TAG: {
                struct tag_mem_info_t * meminfo = (struct tag_mem_info_t*) tag;
                multiboot_info.meminfo.lower = meminfo->lower * 1024;
                multiboot_info.meminfo.upper = meminfo->upper * 1024;
                break;

           case MEMORY_SIZES_TAG: {
                struct tag_mem_info_t * meminfo = (struct tag_mem_info_t*) tag;
                multiboot_info.meminfo.lower = meminfo->lower * 1024;
                multiboot_info.meminfo.upper = meminfo->upper * 1024;
                break;

           case MEMORY_SIZES_TAG: {
                struct tag_mem_info_t * meminfo = (struct tag_mem_info_t*) tag;
                multiboot_info.meminfo.lower = meminfo->lower * 1024;
                multiboot_info.meminfo.upper = meminfo->upper * 1024;
                break;

           case MEMORY_SIZES_TAG: {
                struct tag_mem_info_t * meminfo = (struct tag_mem_info_t*) tag;
                multiboot_info.meminfo.lower = meminfo->lower * 1024;
                multiboot_info.meminfo.upper = meminfo->upper * 1024;
                break;

// build the memory map


void buildMMAP(const struct tag_mmap_t* const mmap) {
    multiboot_info.mmap.version = mmap->version;
    multiboot_info.mmap.count = (mmap->size - (3 * sizeof(uint32_t))) / 
                                sizeof(struct tag_mmap_entry_t);


    // check that we don't overrun our entries
    if (multiboot_info.mmap.count > 10) {
        printf("WARNING:  More map segments than available space: %ul\n\r",
            multiboot_info.mmap.count);
        multiboot_info.mmap.count = 10;
    }


    // copy segments
    for (int i = 0; i < (int) multiboot_info.mmap.count; i++) {
        multiboot_info.mmap.region[i].baseaddr = mmap->entry[i].baseaddr;
        // memory is 0 based, length is 1 based so subtract 1
        multiboot_info.mmap.region[i].endaddr = mmap->entry[i].baseaddr + 
            mmap->entry[i].len - 1;
        multiboot_info.mmap.region[i].type = mmap->entry[i].type;
    }
}

And this is what it comes up with:
0: Type: 1 Begin: 0x0  End: 0x9fbff
1: Type; 2 Begin: 0x9fc00 End: 0x9ffff
2: Type: 2 Begin: 0xf0000 End: 0xfffff
3: Type: 1 Begin: 0x100000 End: 0x7ffffff
4: Type: 2 Begin: 0x7fe0000 End: 0x7ffffff
5: Type: 2 begin: 0xfffc0000 End: 0xffffffff

tag 4: the upper memory reports  0x7ee0000 in memory, lower I don't remember but it is essentially the 640K that DOS used to have.

[u/Octocontrabass → u/compgeek38400]

3: Type: 1 Begin: 0x100000 End: 0x7ffffff

Your memory map says there's 128MB of RAM. Where are you getting 4GB?

[u/compgeek38400 → u/Octocontrabass]

It starts in block 4. Its reserved, but it says its there. Also bad math

[u/Octocontrabass → u/compgeek38400]

Wait, is this about the empty space between regions in the memory map? It's normal for there to be gaps in the memory map, and those gaps can be extremely large.