Blog from December, 2009

Best Productivity Tool

(smile)

Bootstrapping, testing API from inside the Delta Shell

The following will be soon possible to do in Delta Shell:

After running the script, a new time series is created and then added to the project. Then it is opened in the view.

Maybe this is the way we should write our integration tests in the future (smile). It stimulates to have a well designed API. For example I immediately think that adding items to the project and opening them in the gui can be simplified (KISS principle), e.g.:

# create water depth time series, add it to the project and open it in the gui

waterDepth = TimeSeries[float](Name = "water depth")

waterDepth[DateTime(2009, 1, 1)] = 0.1
waterDepth[DateTime(2009, 1, 2)] = 0.2
waterDepth[DateTime(2009, 1, 3)] = 0.5
waterDepth[DateTime(2009, 1, 4)] = 0.3
waterDepth[DateTime(2009, 1, 5)] = 0.4
waterDepth[DateTime(2009, 1, 6)] = 0.1

CurrentProject.Add(waterDepth)

Gui.OpenView(waterDepth)

What is important is to be consistent in the API and to know that it must be stabilized asap, if people start using it - it will be hard to change it. Here are a good tips about it:

How to Design a Good API and Why it Matters by Joshua Bloch

See original including video presentation here: http://www.infoq.com/articles/API-Design-Joshua-Bloch

All programmers are API designers. Good programs are modular, and intermodular boundaries define APIs. Good modules get reused.

APIs can be among your greatest assets or liabilities. Good APIs create long-term customers; bad ones create long-term support nightmares.

Public APIs, like diamonds, are forever. You have one chance to get it right so give it your best.

APIs should be easy to use and hard to misuse. It should be easy to do simple things; possible to do complex things; and impossible, or at least difficult, to do wrong things.

APIs should be self-documenting: It should rarely require documentation to read code written to a good API. In fact, it should rarely require documentation to write it.

When designing an API, first gather requirements—with a healthy degree of skepticism. People often provide solutions; it's your job to ferret out the underlying problems and find the best solutions.

Structure requirements as use-cases: they are the yardstick against which you'll measure your API.

Early drafts of APIs should be short, typically one page with class and method signatures and one-line descriptions. This makes it easy to restructure the API when you don't get it right the first time.

Code the use-cases against your API before you implement it, even before you specify it properly. This will save you from implementing, or even specifying, a fundamentally broken API.

Maintain the code for uses-cases as the API evolves. Not only will this protect you from rude surprises, but the resulting code will become the examples for the API, the basis for tutorials and tests.

Example code should be exemplary. If an API is used widely, its examples will be the archetypes for thousands of programs. Any mistakes will come back to haunt you a thousand fold.

You can't please everyone so aim to displease everyone equally. Most APIs are overconstrained.

Expect API-design mistakes due to failures of imagination. You can't reasonably hope to imagine everything that everyone will do with an API, or how it will interact with every other part of a system.

API design is not a solitary activity. Show your design to as many people as you can, and take their feedback seriously. Possibilities that elude your imagination may be clear to others.

Avoid fixed limits on input sizes. They limit usefulness and hasten obsolescence.

Names matter. Strive for intelligibility, consistency, and symmetry. Every API is a little language, and people must learn to read and write it. If you get an API right, code will read like prose.

If it's hard to find good names, go back to the drawing board. Don't be afraid to split or merge an API, or embed it in a more general setting. If names start falling into place, you're on the right track.

When in doubt, leave it out. If there is a fundamental theorem of API design, this is it. It applies equally to functionality, classes, methods, and parameters. Every facet of an API should be as small as possible, but no smaller. You can always add things later, but you can't take them away. Minimizing conceptual weight is more important than class- or method-count.

Keep APIs free of implementations details. They confuse users and inhibit the flexibility to evolve. It isn't always obvious what's an implementation detail: Be wary of overspecification.

Minimize mutability. Immutable objects are simple, thread-safe, and freely sharable.

Documentation matters. No matter how good an API, it won't get used without good documentation. Document every exported API element: every class, method, field, and parameter.

Consider the performance consequences of API design decisions, but don't warp an API to achieve performance gains. Luckily, good APIs typically lend themselves to fast implementations.

When in Rome, do as the Romans do. APIs must coexist peacefully with the platform, so do what is customary. It is almost always wrong to “transliterate” an API from one platform to another.

Minimize accessibility; when in doubt, make it private. This simplifies APIs and reduces coupling.

Subclass only if you can say with a straight face that every instance of the subclass is an instance of the superclass. Exposed classes should never subclass just to reuse implementation code.

Design and document for inheritance or else prohibit it. This documentation takes the form of selfuse patterns: how methods in a class use one another. Without it, safe subclassing is impossible.

Don't make the client do anything the library could do. Violating this rule leads to boilerplate code in the client, which is annoying and error-prone.

Obey the principle of least astonishment. Every method should do the least surprising thing it could, given its name. If a method doesn't do what users think it will, bugs will result.

Fail fast. The sooner you report a bug, the less damage it will do. Compile-time is best. If you must fail at run-time, do it as soon as possible.

Provide programmatic access to all data available in string form. Otherwise, programmers will be forced to parse strings, which is painful. Worse, the string forms will turn into de facto APIs.

Overload with care. If the behaviors of two methods differ, it's better to give them different names.

Use the right data type for the job. For example, don't use string if there is a more appropriate type.

Use consistent parameter ordering across methods. Otherwise, programmers will get it backwards.

Avoid long parameter lists, especially those with multiple consecutive parameters of the same type.

Avoid return values that demand exceptional processing. Clients will forget to write the specialcase code, leading to bugs. For example, return zero-length arrays or collections rather than nulls.

Throw exceptions only to indicate exceptional conditions. Otherwise, clients will be forced to use exceptions for normal flow control, leading to programs that are hard to read, buggy, or slow.

Throw unchecked exceptions unless clients can realistically recover from the failure.

API design is an art, not a science. Strive for beauty, and trust your gut. Do not adhere slavishly to the above heuristics, but violate them only infrequently and with good reason.

Fixing a hole...Ants memory profiler, IDisposable and memory leaks.

You can use ants memory profiler to find mememory leaks in your .net code and therefore also in your test code. Using the nunit.exe (classic nunit gui) is the best way to do this. A how to:

1) Find a test that consumes memory by looking at the memory consumption of nunit.exe when you run it sequentially.
2) Fire up ants memory profiler and profile nunit.exe
3) Run your test once.
4) Make a snapshot
5) Run your test again
6) Make another snapshot

By comparing the snapshots you can see which objects were added by the 2nd run. How to link these objects to a memory leak is shown in this nice movie: http://www.red-gate.com/products/ants_memory_profiler/overview.htm
Basically it's all about breaking references to never collected (static?) objects.

For me it turned out I had a lot of IDisposable objects (which require you to call Dispose() when you are done) which were not disposed (or the dispose did not clean up all references). By breaking all references to static objects (or other GC-roots) the object will be collected when the garbage man comes around More about disposable here:http://www.eggheadcafe.com/articles/20050625.asp

Build time break down

FYI here's a statistic of the current Delftshell full (debug) solution build process.

Target Performance Summary:
0 ms _GenerateResolvedDeploymentManifestEntryPoint 1 calls
0 ms ResGen 90 calls
0 ms CreateSatelliteAssemblies 90 calls
0 ms PrepareResourceNames 90 calls
0 ms ResolveReferences 90 calls
0 ms CompileRdlFiles 90 calls
0 ms AfterCompileWinFX 88 calls
0 ms _AfterCompileWinFXInternal 88 calls
0 ms BeforeResolveReferences 90 calls
0 ms CreateCustomManifestResourceNames 90 calls
0 ms BuildOnlySettings 90 calls
0 ms Compile 90 calls
0 ms _TimeStampAfterCompile 56 calls
0 ms BeforeResGen 90 calls
0 ms SetWin32ManifestProperties 90 calls
0 ms AfterResGen 90 calls
0 ms AfterCompile 90 calls
0 ms AfterResolveReferences 90 calls
0 ms _SetExternalWin32ManifestProperties 1 calls
0 ms ComputeIntermediateSatelliteAssemblies 44 calls
0 ms _CopySourceItemsToOutputDirectory 90 calls
0 ms BeforeCompile 90 calls
0 ms GenerateManifests 1 calls
0 ms ValidateToolsVersions 1 calls
0 ms CheckAssemblyTime 83 calls
0 ms PrepareResources 90 calls
0 ms _SGenCheckForOutputs 2 calls
0 ms AfterMarkupCompilePass1 88 calls
0 ms PrepareForRun 90 calls
16 ms PostSharp15GetDirectory 26 calls
16 ms _CopyManifestFiles 1 calls
16 ms PostSharp15DefineConstant 26 calls
16 ms GetNativeManifest 50 calls
16 ms GetReferenceAssemblyPaths 90 calls
16 ms ValidateSolutionConfiguration 1 calls
16 ms _CopyOutOfDateSourceItemsToOutputDirectoryAlways 2 calls
47 ms ResolveKeySource 15 calls
47 ms _CheckForCompileOutputs 90 calls
47 ms PrepareRdlFiles 90 calls
62 ms PostSharp15InspectConstants 26 calls
78 ms GetTargetPath 90 calls
78 ms CoreBuild 90 calls
94 ms _SetEmbeddedWin32ManifestProperties 89 calls
94 ms FileClassification 88 calls
94 ms ResolveVCProjectReferences 90 calls
94 ms _GenerateSatelliteAssemblyInputs 90 calls
109 ms PreBuildEvent 3 calls
125 ms _GenerateCompileInputs 90 calls
125 ms EntityDeploy 87 calls
141 ms BeforeBuild 90 calls
172 ms GetFrameworkPaths 90 calls
188 ms _SplitProjectReferencesByFileExistence 90 calls
203 ms SplitResourcesByCulture 90 calls
219 ms _CheckForInvalidConfigurationAndPlatform 90 calls
250 ms AssignTargetPaths 90 calls
250 ms GetWinFXPath 88 calls
266 ms GenerateDeploymentManifest 1 calls
312 ms SplitProjectReferencesByType 85 calls
422 ms PrepareForBuild 90 calls
516 ms PostBuildEvent 6 calls
609 ms PostSharp15InspectReferences 26 calls
672 ms CreateManifestResourceNames 44 calls
828 ms IncrementalClean 90 calls
844 ms _TimeStampBeforeCompile 56 calls
1047 ms GenerateSatelliteAssemblies 6 calls
1078 ms _CopyAppConfigFile 53 calls
1547 ms GenerateApplicationManifest 1 calls
1906 ms GenerateSerializationAssemblies 2 calls
2688 ms _DeploymentComputeClickOnceManifestInfo 1 calls
3672 ms CopyFilesToOutputDirectory 90 calls
6234 ms apps\DelftShellPlugins\Habitat\modules\PcRasterHelper 1 calls
6578 ms CopyExternalDependencies 26 calls
7766 ms CompileLicxFiles 2 calls
9422 ms CoreResGen 90 calls
14234 ms _CleanGetCurrentAndPriorFileWrites 90 calls
14812 ms GetCopyToOutputDirectoryItems 90 calls
14812 ms AfterBuild 90 calls
22078 ms ResolveProjectReferences 90 calls
26000 ms _CopyOutOfDateSourceItemsToOutputDirectory 81 calls
35359 ms CoreCompile 90 calls
38531 ms ResolveAssemblyReferences 90 calls
68062 ms PostSharp15 26 calls
133266 ms _CopyFilesMarkedCopyLocal 90 calls
425172 ms Build 91 calls

Task Performance Summary:
0 ms FormatUrl 3 calls
0 ms PostSharp15CheckPfx 7 calls
0 ms PostSharp15GetCurrentProjectDirectory 26 calls
0 ms Warning 25 calls
0 ms PostSharp15TouchWithDelay 26 calls
0 ms FormatVersion 1 calls
16 ms RequiresFramework35SP1Assembly 1 calls
16 ms ReadLinesFromFile 90 calls
16 ms CreateItem 78 calls
16 ms GetFrameworkSdkPath 90 calls
31 ms ResolveKeySource 15 calls
47 ms Delete 90 calls
47 ms ConvertToAbsolutePath 90 calls
62 ms GetFrameworkPath 179 calls
78 ms FindAppConfigFile 90 calls
78 ms AssignCulture 105 calls
78 ms EntityDeploy 87 calls
78 ms CreateProperty 286 calls
94 ms GetWinFXPath 88 calls
172 ms AssignTargetPath 450 calls
172 ms Message 487 calls
172 ms WriteLinesToFile 86 calls
266 ms GenerateDeploymentManifest 1 calls
297 ms AssignProjectConfiguration 85 calls
438 ms RemoveDuplicates 180 calls
547 ms MakeDir 169 calls
641 ms CreateCSharpManifestResourceName 62 calls
656 ms PostSharp15MoveWithRetry 52 calls
1047 ms AL 6 calls
1500 ms GenerateApplicationManifest 1 calls
1906 ms SGen 2 calls
2672 ms ResolveManifestFiles 2 calls
6234 ms CallTarget 1 calls
6234 ms VCBuild 1 calls
7750 ms LC 2 calls
9328 ms GenerateResource 35 calls
13484 ms FindUnderPath 450 calls
35188 ms Csc 85 calls
38484 ms ResolveAssemblyReference 90 calls
82047 ms Exec 69 calls
163859 ms Copy 711 calls
453734 ms MSBuild 271 calls