If we load a piece of software into a computer, we change the distribution of electrons in its storage chips or the orientation or magnetization of particles in the surface of a hard disk.
We may view learning processes of organisms as physical changes or we may view them as information storage as well. The material changes might consist in changes of the structure or chemistry of synapses, in the formation of new dendrites, axons or synapses, in the formation of new neurons or in the “pruning”, i.e. destruction, of neurons, dendrites, axons or synapses. It might also involve changes to the glial sheaths surrounding the axons, resulting in changes of the speed and timing of signals transmitted along them.
Thus learning processes involve physical modifications of the brain that we may view as changes to the brain’s “hardware” but that we can just as well view as changes of its “software”, i.e. as instances of information storage. There is no real difference here; instead these are two different ways of looking at the same process.
Whatever “technology” is used to store information, if we look at paper, memory chips, or neurons, what we describe as information storage always involves a change in the material structure of the object storing the information. This change causes the object to act and to react differently afterwards. We may then view these changes as “information storage” in an otherwise unchanged object or system. But this view is just a useful abstraction: We divide the system conceptually into the system itself and its memory and we treat the physical changes that happen when information is stored as if they do not affect the “hardware”.
Whatever processes are involved, on a physical layer of description we may view these changes as structural changes; on an informational layer of description we may describe them as the storage of information in the neural network.
Note that such a distinction of different ways of description can also be applied to genes: the replication and transcription of DNA may be viewed as (auto-)catalytic chemical (i.e. physical) processes resulting in the creation or modification of molecules or as processes of information copying. In any case, in computers, nervous systems or in DNA, we can think of such processes as material processes (“hardware”) or as informational processes (“software”).
We can use the terms “hardware” and “software” to distinguish between aspects of the object or system storing or processing information that do not change (these we could call “hardware”) and those that do change (the “software”).
Such categories should be used with care, however. In learning processes of humans or other animals, certain structures might be changed but then become quite fixed. We should, therefore, not treat our categories as something cast in concrete. We should rather view them as part of the software.
(The picture is from https://commons.wikimedia.org/wiki/File:NanoScanHrMFMWD3200BEVT.JPG. It is showing a “A high resolution magnetic force microscopy image of a late generation hard disc (WD3200BEVT). The image is 1 micron square.”)