I can't answer in general, but in probability, square brackets are used to signal notation hacks. It basically means "please don't parse this like usual".
When someone writes $f(something)$, generally what they mean is the following:
- $f$ is a function, with well defined domain and codomain (even if these haven't been mentioned explicitly);
- Whatever is written in place of $something$ is a element from the domain, or a variable expression that can only take on values of the domain (that is, it only "evaluates" to values in the domain after the variables are bound)
A probability measure is a function that takes something from the event space (its domain) and output something in $[0,1]$. At first, it seems you could just stick to regular function notation to talk about it. Let's consider a simple bernoulli trial. Its probability measure is:
$P(a) = p\quad $ when $a=1$
$P(a) = (1-p)\quad $ when $a=0$
However, this gets stale fast. Say you have two random variables, $X$ and $Y$. The values $X$ can take are $S_X$, and the values $Y$ can take are $S_Y$. Your sample space is $S= S_X \times X_Y $. How do you write down the probability of $X$ being, say, between 1 and 2, using only regular set theory notation?
Like this:
$P([1,2] \times S_Y)$
or perhaps:
$P( {\langle x,y \rangle : x \in [1,2]})$
The problem here is that we don't care about the values $Y$ takes, but we have to explicitly mention $S_Y$ it in order to construct valid samples. It would be ideal if our notation didn't make us do it (what if we had 10 variables?). So we just use square brackets and write it like this:
$P[ X \in [1,2]]$
$X\in [1,2]$ looks like a boolean value. It, however, should be understood as a subset from the sample space ("all samples in which $X\in [1,2]$"). If you just wrote $P( X \in [1,2])$, with regular parenthesis, this could be parsed as inputting $true$ or $false$ to $P$, which makes no sense.
$P(X∈[1,2]) = P(true) = \ ? \ $ ($true$ is not in the domain of $P$)
You could argue that I'm being too strict: $P( X \in [1,2])$ is perfectly reasonable if you consider $X\in [1,2]$ as a shorthand for an event. But still, you'd have to break the rules to talk about conditional probability:
$P[A|B] \triangleq \dfrac{P[A\cap B]}{P[B]}$
There's just no way to interpret $A|B$ as an event in this case. $P[A|B]$ could be different from every possible value of the original probability measure function! Consider the following probability measure:
$P(\{1\}) = 1/3$
$P(\{2\}) = 1/3$
$P(\{3\}) = 1/3$
In this case, $P(x)$ can evaluate to 0, 1/3, 2/3 or 1. Now let's consider the events $A = \{1\}$ and $B=\{1,2\}$:
$P[A|B] = 1/2$
Since we generally want this more convenient notation, we just use square brackets all the time for probability measures, as a warning that things might get weird.