Tradecraft Garden ================= (CPL) = Crystal Palace (TCG) = Tradecraft Garden Release 02.02.26 ==== - (CPL) Added reference and license commands to document more project meta-info in a .spec - (CPL) Added a Yara rule generator to generate rules from a Crystal Palace output. The rule generator is aware of Crystal Palace's transformations (+regdance, +shatter, etc.) and generates rules for the islands of invariance--the stuff that didn't change. - Added -g "file.yar" flag to ./link and ./piclink to output yara rules - Added Java API to generate rules - Added rule "name" maxSigs minAgree minSigLen-maxSigLen "optInFuncList, ..." to let .spec files provide optimal rule advice on specific pieces of the project. (Set maxSigs to 0 to opt a component out of rule generation) - (CPL) Crystal Palace now tracks open output file (e.g., disassemble, coffparse) handles globally through the linker session. What this means is you can specify the same filename in your .spec and the information will all get appended/dumped to one file. - (CPL) Added a LinkSpec API getName(String defaultv) to get the .spec set name of the project with a fallback default of your choosing. getName() will derive a default name using the .spec filename still. - (CPL) disassemble (CLI & .spec command) now use a more analyst-friendly output style. Intel-style syntax, memory-sizes explicitly shown, and sane formatting of numbers - (CPL) Added before "command1" : command2 to execute command2 whenever command1 is run. The intent is to allow @config.spec files to insert debug commands (e.g., disassemble) or override the default rule advice without editing the project's .spec files. - (CPL) Added printed warnings during use of deprecated Java APIs in LinkSpec. I plan to remove the deprecated APIs in a future update. - (CPL) Added pop $VALUE to pop stack byte[]s into $VALUE - (CPL) $VARs are now mutable, except for $DLL and $OBJECT when they're already set - (CPL) Added a basic crash handler to demo\run.x86.exe and demo\run.x64.exe. - (CPL) Added options command to add +options to existing "make *" options. - (CPL) Added meta "verb" "value" command to update meta-info for a .spec. This (& options) is mainly useful chained with "before" because it allows updating license/author/etc. from a CLI included @config.spec file. It's how I generated the nice rules below: - (TCG) Added x64 yara rules for pagestream, rdllinjection, and libtcg. With the exception of PicoLoad, I generated each of these with +regdance and +mutate on. Release 01.15.26 ==== - (CPL) Reworked "make coff" so it's linkfunc friendly and link'd data goes to .rdata. Release 01.13.26 ==== - (CPL) Added rudimentary COMDAT folding. Fixes issues with merged modules w/ same globals - (CPL) Made fixbss/fixptrs friendlier to COMDAT globals. - (CPL) Added cmp [imm32], %reg32 to +mutate to break up these constants - (CPL) Refactored the BTF - (CPL) Added +blockparty BTF option to randomize order of code blocks within functions - (CPL) +optimize now trims int3/NOP padding instructions present at the end of some functions - (CPL) Added +shatter BTF option. It's like +blockparty, but it randomizes blocks globally - (CPL) Added +regdance to randomize non-volatile registers in some functions - (TCG) Cleaned up TCG Makefiles and made -O1 the default for most examples. Made a few code mods to support -O1 (e.g., simple_rdll_hook tried to inline and optimize the setupHooks function targeted by redirect, so I needed to work-around that). - (CPL) coffparse and disassemble now throw an error if you attempt to define them twice - (CPL) Merged fixptrs (x86) & fixbss (x86) transforms. More coverage for both. Less fragility. - (CPL) Added fixbss/fixptrs instruction transforms to support MinGW -O1 compiled programs. - (CPL) LTO now walks jumps to functions. - (CPL) fixptrs (x86) & fixbss (x86/s64) will warn if a transform might corrupt flags and bail - (CPL) Added dfr/attach/redirect support for jmp calls. These should only show up in -Os or -O2/-O3 compiled programs. For Crystal Palace PIC -O2/-O3/-Os are not supported, expect bad things. For PICOs -Os is probably tolerable. Use -O1 if you have control over this. - (CPL) Fixed unaligned %rsp fixbss and dfr calls for some transforms (x64) - (CPL) Added "dirty leaves" check to determine if an -O1 optimized function is not aligned. This is used to make sure dfr/fixbss inserted calls are always made with an aligned %rsp - (CPL) Fixed a bug breaking parsing of no argument commands with a # comment on the same line - (CPL) Fixed bug where command parser did not treat a tab as argument separating whitespace - (CPL) Hardened code to derive a memory-operand from a CISC instruction to return null if there's an unexpected register; but to also print details so I can see what's up - (CPL) Added "#!/usr/bin/env bash" to Crystal Palace scripts. Thanks Henri Nurmi for patch - (CPL) Fixed Null Pointer Exception with loader.spec used on CLI from current folder. Again, thanks Henri Nurmi for the patch. - (CPL) Got rid of the "function is not in MODULE$Format" error associated with attach command. - (CPL) Removed CISC, NOP, and reg zeroing mutations from +mutate. This feature will focus on breaking up constants in individual instructions going forward. - (CPL) Updated DangerWalk (safe OutputDebugStringA check) to use LTO's Call Walker. - (CPL) Revised ./link and ./piclink CLI help message. Motivated by a need to emphasize that '$' is optional when specifying a $VAR on the CLI. I didn't like \$. - (CPL) demo\run.x86.exe and demo\run.x64.exe print GPA/GMH/ENVKEY without '$' sigils again - (CPL) Fixed demo\run.x64.exe to pad GPA/GMH pointers with zeroes Release 12.01.25 ==== - (CPL) Added support for DFR resolutions to non-%eax/%rax registers - (CPL) optout check now walks full hook chain (from candidate hook) to detect if an optout function is ever called and rejects the candidate if it is. - (CPL) Added feature to protect/check integrity of EFLAGS/RFLAGS in BTF modifications - (CPL) Made some of the "can't transform" error messages clear about associated feature - (CPL) go() is now optional in PICOs. LTO requires go() or 1 exported function. PicoEntryPoint (in updated LibTCG) will return NULL if go() doesn't exist. This is to allow unrelated PICOs to compose together more easily and expose their capability via exportfunc/PicoGetExport - (CPL) Fixed bug where exportfunc __tag_func's don't propagate up from "run file.spec" - (CPL) Added string %variables to .spec command parser. Unquoted positional arguments starting with % will get evaluated through this mechanism. - (CPL) Added concatenation operator <> to .spec command parser. This concatenates strings and %variables within a command string (e.g., disassemble %name <> ".txt") - (CPL) Added setg "%var" "value" and set "%var" "value" to set a global %variable and a %variable that is local to the current .spec file. - (CPL) run "file.spec" now accepts poisitional arguments and passes them as local variables to the run specification file. Vars are %1 (first arg), %2 (second arg), etc. - (CPL) added echo command to echo its arguments to STDOUT (CLI) or a SpecLogger (API). - (CPL) added : as a "quote everything else into one argument" delimeter in Crystal Palace's command language. It'll work anywhere, but it's mainly for... - (CPL) added foreach "a, b, c": command %_ to run one or more commands over a comma-separated list of values. Remember, you can use %var in place of "a, b, c" - (CPL) added next "%var": command %_ which removes the first element of %var, assigns it to %_, and runs the command. This only happens if %var isn't empty. - (CPL) You can now use %var where you'd use $VAR; but %var must resolve to a '$VAR' string - (CPL) Refactored COFF / DLL parsing & arch extraction into its own Capability class and updated LinkSpec to use this. This is part of the published Java API for Crystal Palace now - (CPL) Changed Crystal Palace Main-Class to crystalpalace.spec.LinkerCLI and updated the link and piclink shell scripts to use it (and preserve quoted arguments properly too) - (CPL) link and piclink CLI now allow \$KEY=#### to set binary data (as before), but now with the $ sigil. %key="value" on CLI sets a variable. @config.spec runs a Crystal Palace .spec as a config file for the project. You may specify multiple $KEYs, %vars, and @config.specs. - (CPL) Deprecated several functions in the LinkSpec Java API in a favor of a LinkSpec and Capability (argument) split. - (CPL) Added callable labels. Declare with name.arch[.o|.dll]: and call w/ .name arg1 arg2 - (CPL) Added call file name arg1 arg2 to call a callable label in another .spec file - (CPL) Added resolve '%files' to immediately walk list of files, canonicalize them relative to the current .spec, and store the result back into %files. - (CPL) Fixed bug that treated full paths as childs path of current .spec folder sometimes - (CPL) Added pack $DEST "template" "arg1" "arg2" ... to pack arguments into a $VAR byte[] - (CPL) '#' is now allowed inside of "quoted strings" w/o being interpreted as a comment char - (CPL) Added MOV r32, r/m32 (not EAX) instruction form to x86 fixptrs logic - (CPL) Added addhook "MODULE$Function" (without a specified hook function). This form will use MODULE$Function hook chain (created with attach) as the wrapper in __resolve_func. What does this mean? attach layering is now possible with IAT hooks created this way. - (CPL) Fixed fixptrs MOV r/m32, imm32 with non-EAX register transform. - (CPL) Added linkfunc "symbol" to link data (presumably PIC) to a function in a PICO or PIC - (CPL) ./piclink now accepts a callable label (name.x86/name.x64) as a target - (TCG) Made Simple RDLL - Hooking modular and split XORHooks and Stack Cutting into modules - (TCG) Made Simple Guardrails agnostic to its stage 2 with these new features. Release 11.10.25 ==== - (CPL) Consolidated dfr, fixptrs, fixbss into a single disassemble, modify, rebuild pass. Part of an architectual effort to allow distinct pass groups of mutually exclusive modifications. - (CPL) Added attach "MODULE$Function" "hook" to attach a hook to calls/references to MODULE$Function in a PIC or PICO. Calls/references to MODULE$Function in hook are not touched. Multiple hooks for one MODULE$Function is OK (so long as each hook calls MODULE$Function) - (CPL) Added redirect "function" "hook" to change calls/references for local function to hook. Similar semantics to attach. References to function within hook AND function are not touched. - (CPL) fixptrs now supports MOV r/m32, imm32 with EAX register - (CPL) fixptrs now supports CMP EAX, imm32 - (CPL) Added preserve "MODULE$Func|symbol" "symbol1, symbol2, ..." to opt specific functions out of attach and redirect instrumentation for the specified target or MODULE$Func symbol - (CPL) Crystal Palace commands that accept a function symbol will now (when possible) suggest a symbol when the specified function doesn't exist. This should aid x86 .specs where the naming conventions (esp. for __stdcall attach hooks) can get a little wonky. - (CPL) Added features to aid GetProcAddress IAT hooking. addhook "MODULE$Function" "_hookSymbol" registers an IAT hook MODULE$function. filterhooks $DLL|$OBJECT filters registered hooks using the import table of $DLL or $OBJECT. And, __resolve_hook(DWORD) is a linker intrinsic to generate inline code to map an ror13 "Function" string hash to a registered hook function. - (CPL) Added protect "func1, func2, ..." to opt functions out of attach/redirect instrumentation. dprintf/_dprintf is automatically in this list. - (CPL) Added danger walk checks to look for dprintf use from fixptrs/fixbss/dfr functions. Why? OutputDebugStringA uses SEH and SEH + these dynamically inserted function calls contexts can cause mem corruptions. These call contexts do a lot of register pushes to protect state and that interferes with the post-exception stack walking heuristics when there's no unwind info - (TCG) Made several (small) changes to LibTCG and tcg.h. I've updated the TCG examples to use these. They'll require some changes to keep your compiler happy too: - Changed function prototypes of findModuleByHash and findFunctionByHash to better align with GetModuleHandle and GetProcAddress. - added adler32sum and ror13hash functions to LibTCG - added __resolve_hook() definition too - (CPL) Added exportfunc "funcsymbol" "__tag_functag" to export a PICO function and associate it w/ a __tag_functag() intrinsic function. Use PicoGetExport in LibTCG to get these function addresses - (TCG) Rewrote Simple Loader (Hooking) and Stack Cutting to demonstrate better modularity practices - (TCG) Slight refactor of Page Streaming to separate DLL loading from page streaming module - (CPL) Added optout "function" "hook1, hook2, etc." to prevent specific attach/redirect hooks in the context of a function. Use this to opt a tradecraft setup out of its own instrumentation without closing the door to layering other tradecraft in to benefit it. - (CPL) Removed reladdr. Use fixptrs to avoid the x86 address hacks. - (CPL) Got rid of implied reladdr "permission" to resolve x86 relocs with partial pointers for data appended to the PIC with link. This effectively makes fixptrs mandatory for x86 PIC that references data appended to it. - (TCG) Updated rdllinjection to use fixptrs - (CPL) Took imul reg, 0 out of code mutator as a reg zeroing option. Release 10.27.25 ==== - (CPL) COFF normalization now skips .rdata$zzz (GCC's ident spam) unless a reloc refers to it - (CPL) Added fixbss "getbss" to restore access to uninitialized global variables in x86/x64 PIC. The getbss function is responsible for returning a pointer to read/write memory for the .bss section. It needs to return the same pointer each time. - (CPL) Fixed DFR "func" "strings" x64 crash by making sure the pushed strings don't break the stack 16b alignment contract. - (CPL) Fix LTO walk bug that would miss function pointer relocations from merged modules - (CPL) Changed missing label/incorrect target exception to a SpecProgramException and it now applies across run/runObject/runDLL - (CPL) Changed the contract of run "file.spec" to work more like a generic include. No need to have the run .spec end by returning bytes[]. The program stack is shared between parent/child script. You get to decide the contract. - (CPL) fixptrs now ignores CALL rel32 relocations. Only time this should come up is with a mistyped/missing function definition. The error message is now clearer "can't resolve reloc" - (CPL) Extended DFR to allow multiple resolvers per PIC with a concept of a default resolver and resolvers that apply to specific modules. dfr "func" "method" "module1, module2" will apply to any imports for module1/module2. dfr "resolver" "method" will set a default resolver--which is only used when no module-specific resolver applies. - (TCG) Added Simple PIC 9 to demonstrate running a PICO as PIC - (CPL) fixptrs is now diligent about saving ecx/edx registers. My caller function doesn't clobber these... but yours might - (CPL) Added remap "old_symbol" "new_symbol" to change a symbol name in current COFF - (TCG) Made Simple Loader 8 simpler with new remap feature (and changes to run) - (CPL) Fixed bug preventing propagation of local func x64 relocations through BTF. This same fix also restores (unresolved) local func x64 symbol names to disassembly output - (CPL) Added common func symbol existence/is a func checks to fixptrs/fixbss values Release 10.13.25 ==== - (CPL) Added Dynamic Function Resolution for x86/x64 PIC! Declare a resolver in your .spec with dfr "resolver" "method" and CPL will rewrite MODULE$Func references to call resolver with either (method=ror13) the ROR hash of MODULE and Func as args or MODULE and Func stack strings as args (method=strings). No more looking up/calculating ROR hashes to use Win32 APIs. - (CPL) Code Mutator now mutates PUSH imm32 (x86) - (CPL) Added BTF transforms to fix x86 pointers to make it more x64 PIC-like. Use Use fixptrs "_function" to specify a function that returns its return address. This opts an x86 program into these transforms. - (CPL) Added mergelib "/path/to/lib.zip" to merge a Crystal Palace library into the PIC/PICO/COFF at the top of the atack. A Crystal Palace library is just a .zip with objects in it. - (CPL) Deprecated "make pic64" -- "make pic" is now the same functionality - (CPL) Added +gofirst PIC option to move go() to position 0 of the program - (TCG) Rewrote the Tradecraft Garden examples to use these new conventions - (CPL) Code Mutator now mutates MOV imm64 %reg (x64) Release 10.13.25 ==== - (TCG) Released Tradecraft Garden examples and projects under the 3-clause BSD license. The code and headers did not change otherwise. https://tradecraftgarden.org/download/tcg20250910-bsd.tgz Release 09.10.25 ==== - (CPL) Rewrote the COFF parser and intermediate representation - (CPL) Added a normalization pass to merge section groups in a COFF before carving data from it - (CPL) Rewrote pic/pic64/object exporters. Functionality should be the same, BUT, PicoCodeSize() no longer rounds up to nearest page. - (CPL) Added 'make coff' to have export output a COFF file - (CPL) Added reladdr "_symbol" to give Crystal Palace permission to resolve relocations for a symbol with a relative address, rather than a direct address. This is necessary for x86 PIC, where one might want to resolve _go to the beginning of the .text section. reladdr symbols require special care in an x86 program to turn into a valid pointer. - (TCG) Updated TCG projects to use reladdr where needed, primarily for _go symbol - (CPL) Added support for IMAGE_REL_AMD32_REL32_1/2/3/4/5 relocations - (CPL) Added coffparse "out.txt" to dump normalized COFF string rep to out.txt - (CPL) Added 'merge' command to merge COFF file (bytes) with object on stack - (TCG) Updated picorun.h to accommodate new loader directives for handling data relocations. The API is the same, but you will want to recompile projects with this updated file. - (TCG) Fixed code mutator bug breaking up an Integer.MIN_VALUE constant Release 07.09.25 ==== - (CPL) Added a link-time optimizer. Use make [type] +optimize (in spec file) to enable - (CPL) Added function "disco": randomizes function order. make [type] +disco enables. - (CPL) Added code mutator. make [type] +mutate enables - (CPL) Added error check for jump tables for PIC/PIC64/PICOs--these are not supported and will cause a crash. - (CPL) make pic/make pic64 warn if "go" (presumed entry point, when it exists) isn't first in the program. - (TCG) Changed function declaration of "go" in PIC to include no_reorder attribute to try and mitigate func. re-ordering when compiled with optimizations on - (CPL) Added to relocation string representation. Gives a better sense of where in program a linking error came from. - (CPL) Added disassemble "file.txt" command to spec file; outputs disassembled PIC/PIC64/PICOs - (CPL) Added ./disassemble [+options] [file.o] command to Crystal Palace, similar to above. - (CPL) Fixed typo in ./coffparse help output. - (CPL) Updated LinkSpec APIs (and CLI) to accept a COFF (PICO) or DLL argument to apply spec to - (CPL) Added test.x86.o and test.x64.o demo programs (Hello World message box) to demonstrate applying a Crystal Palace .spec to a COFF capability - (TCG) Added Simple Loader 7 - COFF Capability to demonstrate pairing COFF with a PIC loader - (TCG) Added Simple Loader 8 - COFF and DLL Capability to demonstrate a project capable of loading either a DLL or COFF. - (TCG) Added #error to resolve_eat.h if WIN_X86 or WIN_X64 not defined. Release 06.04.25 === - Initial release