[DragonFlyBSD - Bug #3114] Using malloc(size_max) gives strange results

[bugtracker-admin at leaf.dragonflybsd.org]

Issue #3114 has been updated by ddegroot.

File nmalloc.c.diff added

This seems to resolve the issue. But i am not quite sure if this is the right way.

Note: in _slaballoc size is not checked for overflow (i think)
Note: in _slaballoc size is rewritten and it might be better to use a seperate local variable instead

```
diff --git a/lib/libc/stdlib/nmalloc.c b/lib/libc/stdlib/nmalloc.c
index b39aaf301..d9bc90fb8 100644
--- a/lib/libc/stdlib/nmalloc.c
+++ b/lib/libc/stdlib/nmalloc.c
@@ -753,6 +753,8 @@ zoneindex(size_t *bytes, size_t *chunking)
        return(0);
 }
 
+#define MUL_NO_OVERFLOW        (1UL << (sizeof(size_t) * 4))
+
 /*
  * malloc() - call internal slab allocator
  */
@@ -761,6 +763,11 @@ __malloc(size_t size)
 {
        void *ptr;
 
+       if ((size >= MUL_NO_OVERFLOW ) || (SIZE_MAX < size)) {
+               errno = ENOMEM;
+               return(NULL);
+       }
+
        ptr = _slaballoc(size, 0);
        if (ptr == NULL)
                errno = ENOMEM;
@@ -769,8 +776,6 @@ __malloc(size_t size)
        return(ptr);
 }
 
-#define MUL_NO_OVERFLOW        (1UL << (sizeof(size_t) * 4))
-
 /*
  * calloc() - call internal slab allocator
  */
@@ -982,6 +987,9 @@ _slaballoc(size_t size, int flags)
                bigalloc_t big;
                bigalloc_t *bigp;
 
+               if ((size >= MUL_NO_OVERFLOW ) || (SIZE_MAX < size) ) {
+                       return(NULL);
+               }
                /*
                 * Page-align and cache-color in case of virtually indexed
                 * physically tagged L1 caches (aka SandyBridge).  No sweat
@@ -989,7 +997,8 @@ _slaballoc(size_t size, int flags)
                 *
                 * (don't count as excess).
                 */
-               size = (size + PAGE_MASK) & ~(size_t)PAGE_MASK;
+               size = (size + PAGE_MASK) & ~(size_t)PAGE_MASK;         /* Note: Changing size, without checking overflow.
+                                                                          also might be better to use a different variable instead of the original request size */
 
                /*
                 * If we have overflown above when rounding to the page
```

----------------------------------------
Bug #3114: Using malloc(size_max) gives strange results
http://bugs.dragonflybsd.org/issues/3114#change-13365

* Author: ddegroot
* Status: New
* Priority: Normal
* Assignee: 
* Category: Userland
* Target version: 5.0.0
----------------------------------------
While porting d-lang dmd/druntime/phobos to DragonFlyBSD, it became apparent that using 'malloc(size_t)' to deduce malloc and alignment rules, gave some unexpected results.

Example:
malloc size:9223372036854775807, malloc failed, ptr == NULL, errno:12                       // expected result (INTPTR_MAX)
malloc size:72036854775808,  ptr == 0x800800000                                             // this is fine
malloc size:18446744073709551613,  ptr == 0x800455000                                       // unexpected UINTPTR_MAX / SIZE_MAX

Related dlang:druntime PR: https://github.com/dlang/druntime/pull/1999

---Files--------------------------------
test_malloc.c (1.01 KB)
test_malloc_results.txt (1.97 KB)
nmalloc.c.diff (1.81 KB)


-- 
You have received this notification because you have either subscribed to it, or are involved in it.
To change your notification preferences, please click here: http://bugs.dragonflybsd.org/my/account