Magic is one of the defining characteristics of fantasy. Whether you are looking at “high” or “low” magic, it’s going to be in there somewhere (I’m pretty sure that fantasy without magic is historical fiction or some other genre). That said, the treatment of magic sets a tone for world-building and how you are “allowed” to use magic in your story.
You can break the division of magic many ways, but for this discussion I’m going to divide magic into whimsical or mechanical magic.
Whimsical magic should bring a sense of wonderment or dread. It should be magic that is incomprehensible in function, if not in effect. It is magic that defies science and logic, which may or may not be portrayed an antithetical to magic.
Why you would use whimsical magic in your story: Largely for the sense of wonderment and awe. You want your protagonist to be constantly off guard, never knowing the limits of what might happen next.
Hidden Rule 1: Your protagonist (and any other POV character) cannot understand how the magic works. They may learn rules for specific interactions or prohibitions (don’t feed the mogwai after midnight, the mushrooms can make you grow taller, etc.), but never gain any depth of understanding of why or how.
Hidden Rule 2: Whimsical magic should never be fully under control. There is a primordial aspect to something that is beyond understanding. It makes for a good adversary (or tool of an adversary) or at best an uncertain ally.
Hidden Rule 3: Beware of deus ex machina. When magic has few rules, the story itself needs to enforce this one. You, as a writer, simply cannot pull a new magic out at the end to solve your heros’s problems. The solution should come either from within the protagonist (love, perseverance, courage, etc.), or through the use of a magic that has already been shown to the reader – and it must work the way it has been known to work.
How to use whimsical magic: You want to put your protagonist outside their comfort zone and by a large margin. You might also want to create a world with a certain aesthetic without bogging yourself down in figuring out how it could be possible. Just be careful that in creating a world without knowing the rules that you don’t also create a world that isn’t believable. When you don’t have a set of rules, continuity can be a tricky matter, especially if you don’t keep track of how things have been treated early in the story versus late. It’s a pitfall that you can climb back out of, just be careful of falling in the first place.
Mechanics (alternate physics)
The other way to look at magic is to treat it somewhat like an alternate set of physics. There are rules, cause and effect, limitations. Mechanical magic is useful for creating a believable, relatable world, even if it’s one that is different from our own.
Hidden Rule 1: Not everyone has to know the rules. There can be practitioners of magic who know incantations and potion mixtures or whatever other forms of magic you devise, without ever understanding what they are doing. This is fine. Ignorance within the narrative is fine, so long as you know the mechanics and limitations of the magic system you create, and apply it accordingly.
Hidden Rule 2: If there are rules, readers are going to want to know them. You don’t have to create a treatise on the subject, but cluing in readers on how magic works in your world can be a great treat for them. One way to accomplish this is for one of your POV characters to be introduced to magic use for the first time. Another way is to have circumstances align such that there needs to be a discussion of why something occurs, and frame the debate around the ways of magic, giving the reader a more oblique perspective.
Hidden Rule 3: Don’t break your own rules. Once you’ve described how magic works, you tread a fine line when you change something. Treat such an incident too lightly and you’ll create dissonance in the story. If the workings of magic change, practitioners should be at least curious as to why, if not outright alarmed. It could signal to them that there is something deeply wrong in the world or hint that their understanding was flawed at a fundamental level.
How to use mechanical magic: Treat mechanical magic as an addition set of physical laws in world-building, and the understanding of them as an additional field of science and technology (possibly even the only field of science and the only real technology). Base the development of the world around these powers and abilities, the ways that the magic would affect not only the world, but the races and cultures within it.
The Magic of Firehurler
The magic system of the Twinborn Trilogy is mechanical. Aether is an actual substance that coexists in an alternate dimension co-located with the physical world. It is both a building material and power source for use in magic. Aether is produced by living creatures, each of which has a Source of aether within it. I go into more detail in my blog about the Twinborn Trilogy’s magic system.