KermMartian wrote:
One thing that CompyNerd has been suggesting repeatedly is that shells offer a common memory management system, like malloc(), which it sounds like is essentially what you're describing. I think that could be a very good way to do things, rather than assume you can always put your data in the same place. Among other things, this opens up the possibility that shells could add task-switching without requiring much or any changes to programs, for example.
Basically that, yep.

Most of the ASM coding I have done has been for x86 and DOS, and DOS uses somewhat of a similar but a bit more advanced system based on a chain of Memory Control Blocks (basically allocated chunks of memory with headers, connected in a chain instead of being indexed in a fixed table). I don't really think DOS has ways or standards of defragmenting the memory pool after it's been used a lot, but it doesn't really matter then since PCs clear RAM on reboot.

DOS's method works wery well if you have a LOT of memory in the pool, but for the small amounts of free RAM in the TI-84, I would say a table would be better, and as I mentioned, defragmentation MUST be a standarized feature of the system. When the pool is fragmented, and a program asks for a chunk of such size that no free memory chunk of suficcient size can be found, the shell must refragment and retry once.

The way I imagine refragmenting can be done, is to make the shell first disable interrupts, then defragment, then issue updated chunk address to all programs that has some chunks of the pool (address to the routine for updating the chunk pointer may be defined in the header of the programs), then restore the interrupt setting, and finally continue/return to what it was previously doing.
KermMartian wrote:
olav_nordmann wrote:
In the most simplest form, the shell only needs to keep track of the memory pool and a table of how it's allocated. The problem is then of course fragmentation, and in situations a program may eventually quit without it's allocated memory being unallocated (say, the program crashes for example). There must be systems in place that looks for those things, and performs garbage collecting or defragmentation of the pool whenever necessary.
One thing that Compynerd has been suggesting repeatedly is that shells offer a common memory management system, like malloc(), which it sounds like is essentially what you're describing. I think that could be a very good way to do things, rather than assume you can always put your data in the same place. Among other things, this opens up the possibility that shells could add task-switching without requiring much or any changes to programs, for example.

Thanks for bringing that up again. Smile Of course, if we don't want to worry about any safeRAM areas at all, programs have to have static pointers to this memory through SMC or the stack. Also, how we implement this system depends on whether we want to allow many allocations or just one per program. I'm leaning toward as many allocations as we want because then DCS's libraries and SEs can also use this, which would lean further towards the dynamic memory assembly idea I was talking about because then the libraries could safely self-modify.
A couple thoughts. First, shell assembly programs that can also be run from the homescreen should not show up on the PRGM menu at all. They can be kept as AppVars all the time. Similarly, using programs as data files should also be deprecated. Thus, we can reserve the PRGM menu for programs that can actually be run from the homescreen.

Second, instead of using the regular assembly header, we can use a different header. I propose starting with D3 3F, which is the BASIC Return token followed by a newline. Thus, even if the shell-based program appears as a regular program, attempting to run a it with either do nothing or throw an error.

Third, it would be nice to include more metadata and bigger icons. Both these use more space. And we have lots more archive space! But we need not copy any metadata into RAM when we run the program, so the .org should specify the start of executable code, not metadata, as the start of the program.

Fourth, tr1p1ea has some idea about screen modes and color modes. But, aside from BGR mode and 16-/18-/fake 24-bit color modes, the LCD controller has no modes, so it's kind of a moot point. Icons and other basic graphics that don't need fancy color depth might use a standard 4-bit palette to save space.

Finally, in regard to the above comments on dynamic memory allocation: DMA is often see as unwieldy on the Z80. However, it could be done by having programs include relocating information in their header and different segments for Code, Initalized Data, and Uninitalized Data. And as long as we make programs relocatable, we might as well include multitasking.


Code:
; Metadata format: The high four bits are the ID, the low 12 bits are the length.
; If the high bit is set, it's a shell-specific or program-specific field.
; Strings are standard ASCII, zero-terminated, and shall never exceed 252 characters in length.
; The normal metadata format is
;     .db lengthLow
;     .db lengthHigh | mdFieldId
; So do something like
;     ld e, (hl) \ inc hl \ ld d, (hl) \ dec hl \ ld a, d \ and mdFieldIdMask
; to get the field type.  If it's not the field you want,
;     ld a, d \ and ~mdFieldIdMask \ ex de, hl \ add hl, de
; will skip to the next field.
; This metadata format can also be used for data files. Now, isn't that nice?
; If a metadata field type appears more than once, it means the field has more than one value.
; For example, a program could have more than one author.
; Although, for some fields, like PrettyName, you might simply only display the first field found.
; Or, multiple fields could specify different languages in a shell-specific manner.
; Finally, if you want icons for data files,
; it's probably a good idea to use ShellType to identify the file type and icon,
; rather than including an icon in every file.
mdFieldIdMask      equ   0F000h
mdMetatdataField   equ   00h
; The metadata block MUST end with mdNoMoreField, i.e. .db 2, 0
mdNoMoreFields      equ   0
mdNoMoreFieldsSize   equ   2
; Shell type is the only REQUIRED field.
; It MUST be at least one byte, but may be up to 253 bytes.
; This field can used not only to identify a specific shell,
; it can also identify what type a data file is.
; In other words, for data files, it may also contain information for launching an editor.
mdShellType      equ   10h
mdShellTypeLength   equ   3
mdPrettyName      equ   20h
mdPrettyNameLength   equ   2
mdDescription      equ   30h
mdDescriptionLength   equ   2
mdAuthor      equ   40h
mdAuthorLength      equ   2
mdVersion      equ   50h
mdVersionLength      equ   2
mdOtherMetadata      equ   60h
mdOtherMetadataLength   equ   3
mdIcon         equ   70h
mdIconLength      equ   4
mdShellSpecific      equ   80h
; Standard fout-bit palette for basic graphics
; The shell routines that render anything using this palette not be particularly fast.
stdBlack      equ   0
stdRed         equ   1
stdGreen      equ   2
stdBlue         equ   4
stdYellow      equ   (stdRed | stdGreen)
stdMagenta      equ   (stdRed | stdBlue)
stdCyan         equ   (stdGreen | stdBlue)
stdGray         equ   (stdReg | stdGreen | stdBlue)
stdBrightColor      equ   8
stdTransparent      equ   8   ; Who needs two shades of gray?
stdPink         equ   (stdBrightColor | stdRed)
stdLightGreen      equ   (stdBrightColor | stdGreen)
stdLightBlue      equ   (stdBrightColor | stdBlue)
stdLightYellow      equ   (stdBrightColor | stdYellow)
stdLightMagenta      equ   (stdBrightColor | stdMagenta)
stdLightCyan      equ   (stdBrightColor | stdCyan)
stdWhite      equ   (stdBrightColor | stdWhite)
HstdBlack      equ   0
HstdRed         equ   (stdRed << 4)
HstdGreen      equ   (stdGreen << 4)
HstdBlue      equ   (stdBlue << 4)
HstdYellow      equ   (stdYellow << 4)
HstdMagenta      equ   (stdMagenta << 4)
HstdCyan      equ   (stdCyan << 4)
HstdGray      equ   (stdGray << 4)
HstdTransparent      equ   (stdTransparent << 4)
HstdPink      equ   (stdPink << 4)
HstdLightGreen      equ   (stdLightGreen << 4)
HstdLightBlue      equ   (stdLightBlue << 4)
HstdLightYellow      equ   (stdLightYellow << 4)
HstdLightMagenta   equ   (stdLightMagenta << 4)
HstdLightCyan      equ   (stdLightCyan << 4)
HstdWhite      equ   (stdWhite << 4)

   .org   userMem - (MainEntryPoint - MetaData)
MetaData:
   .db   tStop, tNewline
   .dw   metadataLength
; The shell should parse these fields and allow them to appear in any order.
   .db   mdShellTypeLength
   .db   mdShellType
   .db   mdShellTypeId   ; Actual shell ID, e.g. MirageOS or whatever
   .db   mdPrettyNameLength + _++ - _+
   .db   mdPrettyName
_:   .db   "Tetris", 0
_:
   .db   mdScriptionLength + _++ - _+
   .db   mdDescription
_:   .db   "A classic game!", 0
_:
   .db   mdAuthorLength + _++ - _+
   .db   mdAuthor
_:   .db   "DrDnar", 0
_:   
   ; Maybe I'm too lazy to supply a version
   ; If a metadata field is not supplied, programs should intelligently
   ; simply consider the field null, instead of behaving strangely.

; WORST ICON EVER
myIconSize = myIconEnd - myIconStart
myIconHeight = 4
myIconWidth = 4
myIconStart:
   .db   mdIcon | (myIconSize >> 8)
   .db   myIconSize & 0FFh
   .db   ((myIconHeight / 4) << 4) | (myIconWidth / 4)
   .db   HstdGray | stdGray, HstdGray | stdGray, HstdGray | stdGray, HstdGray | stdGray
   .db   HstdGray | stdGray, HstdGray | stdGray, HstdGray | stdGray, HstdGray | stdGray
   .db   HstdGray | stdGray, HstdGray | stdGray, HstdGray | stdGray, HstdGray | stdGray
   .db   HstdGray | stdGray, HstdGray | stdGray, HstdGray | stdGray, HstdGray | stdGray
myIconEnd:
   .db   mdNoMoreFieldsLength
   .db   mdNoMoreFields
; END OF METADATA
MainEntryPoint:
   ; Your program actually begins here

DrDnar wrote:
A couple thoughts. First, shell assembly programs that can also be run from the homescreen should not show up on the PRGM menu at all. They can be kept as AppVars all the time. Similarly, using programs as data files should also be deprecated. Thus, we can reserve the PRGM menu for programs that can actually be run from the homescreen.
That sounds rather user-unfriendly to me. If users can't pick ASM programs that can run with something like Doors CS's HomeRun feature, they're not going to realize that they can run those programs at all, I think. Perhaps I misunderstand you.

Quote:
Second, instead of using the regular assembly header, we can use a different header. I propose starting with D3 3F, which is the BASIC Return token followed by a newline. Thus, even if the shell-based program appears as a regular program, attempting to run a it with either do nothing or throw an error.
I think that would be an excellent idea.

Quote:
Third, it would be nice to include more metadata and bigger icons. Both these use more space. And we have lots more archive space! But we need not copy any metadata into RAM when we run the program, so the .org should specify the start of executable code, not metadata, as the start of the program.
Indeed, and it's sort of silly that we didn't do this before.

Quote:
Fourth, tr1p1ea has some idea about screen modes and color modes. But, aside from BGR mode and 16-/18-/fake 24-bit color modes, the LCD controller has no modes, so it's kind of a moot point. Icons and other basic graphics that don't need fancy color depth might use a standard 4-bit palette to save space.
That was my thought for DCS: 4-bit icons for BASIC and ASM programs to keep icon sizes sane.

Quote:
Finally, in regard to the above comments on dynamic memory allocation: DMA is often see as unwieldy on the Z80. However, it could be done by having programs include relocating information in their header and different segments for Code, Initalized Data, and Uninitalized Data. And as long as we make programs relocatable, we might as well include multitasking.
If the memory allocation is coarse-grained, then I don't think it would be much of a problem. The key would be that a program asks for a nice big chunk for itself at the beginning, and it's up to the shell to find a place for it. For example, ask for 768 bytes, and you might get the AppBackupScreen (or equivalent), but you might also not if that's being used for something. It shouldn't matter where that memory is to your program.

Quote:
[metadata format]
Is this some other device's existing format, or a proposal from you?
I'm not so sure about the D33F idea, that's something that you can easily type in the program editor, after all. Unless you want to make it only show edit-protected programs in the shells, which might be fine.
calc84maniac wrote:
I'm not so sure about the D33F idea, that's something that you can easily type in the program editor, after all. Unless you want to make it only show edit-protected programs in the shells, which might be fine.
My other qualm about this is I think it's better if ASM programs run improperly do something specific, like make the OS show an error message, rather than silently end. It's more user-friendly to see an error you can look up (to realize you need a shell) than have the program just display Done and make users think the program is corrupt, or worse, just badly-written.
KermMartian wrote:
DrDnar wrote:
A couple thoughts. First, shell assembly programs that can also be run from the homescreen should not show up on the PRGM menu at all. They can be kept as AppVars all the time. Similarly, using programs as data files should also be deprecated. Thus, we can reserve the PRGM menu for programs that can actually be run from the homescreen.
That sounds rather user-unfriendly to me. If users can't pick ASM programs that can run with something like Doors CS's HomeRun feature, they're not going to realize that they can run those programs at all, I think. Perhaps I misunderstand you.

No, you're right. I frankly wasn't thinking about shells being able to hook into the homescreen.

KermMartian wrote:
calc84maniac wrote:
I'm not so sure about the D33F idea, that's something that you can easily type in the program editor, after all. Unless you want to make it only show edit-protected programs in the shells, which might be fine.
My other qualm about this is I think it's better if ASM programs run improperly do something specific, like make the OS show an error message, rather than silently end. It's more user-friendly to see an error you can look up (to realize you need a shell) than have the program just display Done and make users think the program is corrupt, or worse, just badly-written.

How about D3 2D 3F 3F EF 69? That's Stop!\n\ntAsm84CCmp. It's much more clear in the BASIC editor if the program is accidentally unprotected, yields the ERR: Syntax not doing Asm( on an asm program yields, and has a token BASIC programs cannot have.

Edit: Or D3 BB 6C 3F EF 69 = StopAsmPrgm\ntAsm84CCmp
Oooh, I like that latter suggestion a lot. The Stop! is particularly clever to get BASIC programmers' (and users') attention. And I'm glad that there was no misunderstanding on the former point. I like the out-of-the-box thinking of putting programs in AppVars, though.
KermM wrote:
I like the out-of-the-box thinking of putting programs in AppVars, though.


uhh, mimas files? Ignition Programs?
LuxenD wrote:
KermM wrote:
I like the out-of-the-box thinking of putting programs in AppVars, though.


uhh, mimas files? Ignition Programs?
I don't even know what Ignition is, and your Mimas example is source code, not executable programs. Everyone ever has used PrgmObj and ProtPrgmObj (ie, programs and locked programs) for actual executable programs.
Yeah the screen modes idea was before LCD information was known ... was kind of hoping it had some 1/2, 1/4 scaling modes or something but ahh well Sad.

I do like the ideas put forward to handle 84C executables. I think if we can decide this kind of stuff before the release of new tools (not that they're really needed but still, a good opportunity) it would be beneficial.
Quote:
Or D3 BB 6C 3F EF 69 = StopAsmPrgm\ntAsm84CCmp

^I like this idea, that's actually quite clever. Smile
I like the idea of a polyglot header, but let's step it up a notch.


Code:

OpenLib("ShellName
{S,H,E,L,L,H,E,A,D,E,R,I,N,F,O,
ExecLib
Return
Asm84CPrgm
...


Where the shell header info contains an icon format code, a fixed length of data for that icon format, and a description followed by a null terminator. Alternatively, if a String with a NULL ('\0') token can be parsed, you could pass the data as a String rather than as a list (or use Pascal-style strings, or just conclude that any data after the icon is for the description and let your Ans reading routine do some subtraction).


There's no reason not to implement ExecLib since it is 84+ only, and this means you don't need a homescreen hook for HomeRun style functionality. All assembly programs are immediately nostub.


[edit]
Heck, it can just be a Return token and not a Stop, which allows BASIC programs executing ASM programs to fail gracefully.
Art_of_camelot wrote:
Quote:
Or D3 BB 6C 3F EF 69 = StopAsmPrgm\ntAsm84CCmp

^I like this idea, that's actually quite clever. Smile
We should wait for a few more programmers to weigh in, but I wouldn't be surprised if we get a lot of support for that header.

Elfprince: Hmm, that didn't even occur to me; let me try to get my head around that proposal. I must honestly admit that I don't really understand ExecLib and OpenLib too well.
elfprince13 wrote:
I like the idea of a polyglot header, but let's step it up a notch.


Code:

OpenLib("ShellName
{S,H,E,L,L,H,E,A,D,E,R,I,N,F,O,
ExecLib
Return
Asm84CPrgm
...


Where the shell header info contains an icon format code, a fixed length of data for that icon format, and a description followed by a null terminator. Alternatively, if a String with a NULL ('\0') token can be parsed, you could pass the data as a String rather than as a list (or use Pascal-style strings, or just conclude that any data after the icon is for the description and let your Ans reading routine do some subtraction).


There's no reason not to implement ExecLib since it is 84+ only, and this means you don't need a homescreen hook for HomeRun style functionality. All assembly programs are immediately nostub.


[edit]
Heck, it can just be a Return token and not a Stop, which allows BASIC programs executing ASM programs to fail gracefully.

I think this is one of the best ideas I've heard yet. However, it would mean that the program would only run in exactly one shell of a certain name.
I think USB8x is the only app the really supports it. Might be a good place to look first Smile

[edit]

calc84maniac: it means the program would only run *from the homescreen* with a default app (but could be opened from within an app by anyone who thought they could support it). It wouldn't be that hard to rewrite the header to change the default application either. We would just have to mandate a certain amount of newline padding bytes after the OpenLib to ensure there was room for any valid library name to be written in.
elfprince13 wrote:
I think USB8x is the only app the really supports it. Might be a good place to look first Smile

[edit]

calc84maniac: it means the program would only run *from the homescreen* with a default app (but could be opened from within an app by anyone who thought they could support it). It wouldn't be that hard to rewrite the header to change the default application either. We would just have to mandate a certain amount of newline padding bytes after the OpenLib to ensure there was room for any valid library name to be written in.


Hmm yes, I suppose so. Also, I don't think there's anything stopping it from changing the size of the program to change the ExecLib name either, since the header won't be loaded to the execution address anyway.
olav_nordmann wrote:


DOS's method works wery well if you have a LOT of memory in the pool, but for the small amounts of free RAM in the TI-84, I would say a table would be better, and as I mentioned, defragmentation MUST be a standarized feature of the system. When the pool is fragmented, and a program asks for a chunk of such size that no free memory chunk of suficcient size can be found, the shell must refragment and retry once.


I actually ran into a similar problem at work with the free() function in C not working on memory limited systems. There's a fairly simple way to handle memory fragmentation if you're willing to accept some bookkeeping overhead: Indirection.

Rather than returning a raw pointer to memory, a malloc-like routine can return a pointer to a pointer to the memory location (**ptr). Remember that any system that can defragment must keep a list of allocated blocks in order to do so and effectively, all you need to do is return a pointer to the area's entry in the allocation table, leaving you with no net overhead and allowing significant freedom in defragmentation. The downside is that all memory accesses must run through two memory retrievals, which means defragmentation should really only happen when allocations are asked for.

EDIT: I have some C code I can post later if anyone wants it.
Qwerty.55 wrote:
olav_nordmann wrote:


DOS's method works wery well if you have a LOT of memory in the pool, but for the small amounts of free RAM in the TI-84, I would say a table would be better, and as I mentioned, defragmentation MUST be a standarized feature of the system. When the pool is fragmented, and a program asks for a chunk of such size that no free memory chunk of suficcient size can be found, the shell must refragment and retry once.


I actually ran into a similar problem at work with the free() function in C not working on memory limited systems. There's a fairly simple way to handle memory fragmentation if you're willing to accept some bookkeeping overhead: Indirection.

Rather than returning a raw pointer to memory, a malloc-like routine can return a pointer to a pointer to the memory location (**ptr). Remember that any system that can defragment must keep a list of allocated blocks in order to do so and effectively, all you need to do is return a pointer to the area's entry in the allocation table, leaving you with no net overhead and allowing significant freedom in defragmentation. The downside is that all memory accesses must run through two memory retrievals, which means defragmentation should really only happen when allocations are asked for.

EDIT: I have some C code I can post later if anyone wants it.


Perhaps yet another method would be to have the caller pass a reference to a pointer variable in which to store the pointer, and the system would use that reference when relocating. This would reduce a pointer lookup back to a single 16-bit load.

Calling code example:

Code:
  ld hl,size
  ld de,var
  call malloc
  ;Check output flags for success/failure
  ....
 
  ;Use the allocated memory
  ld hl,(var)
  add hl,de
  ld a,(hl)
  ....
The only issue I see with that is when the variable changes is reused without telling the system via free(). The system would then modify memory locations when the programmer isn't using it.

However, malloc() without free() is pretty much universally an error, so I suppose that's acceptable behavior and it moves the indirection to the memory manager, as you mentioned on IRC.
  
Register to Join the Conversation
Have your own thoughts to add to this or any other topic? Want to ask a question, offer a suggestion, share your own programs and projects, upload a file to the file archives, get help with calculator and computer programming, or simply chat with like-minded coders and tech and calculator enthusiasts via the site-wide AJAX SAX widget? Registration for a free Cemetech account only takes a minute.

» Go to Registration page
Page 2 of 4
» All times are UTC - 5 Hours
 
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

 

Advertisement